Add docker support

- Allow setup script to get credentials from flags
- Add Dockerfile and docker_entrypoint
- Edit README for forked version

.gitignore

@@ -1,8 +1,13 @@
 docs/build/
 __pycache__
+feedlist
+data
 *.swp
 *.pyc
 *.db
 *.ini
 *.txt
 *.bck
+*.png
+*.yml
+.env

AUTHORS

@@ -1,2 +1,5 @@
 Antoine Beaupré <anarcat@debian.org>
-Carl Chenet <chaica@ohmytux.com>
+Carl Chenet <carl.chenet@ohmytux.com>
+Alexis Metaireau <alexis@notmyidea.org>
+The Dod <https://social.weho.st/@thedod>
+Matthias Henze

CHANGELOG

@@ -1,3 +1,84 @@
+## [0.17] - 2021-03-26
+### Changed
+- fix install bug while installing from sources
+- bump changelog
+
+## [0.16] - 2020-12-09
+### Added
+- scripts/register_feed2toot_app: --client-credentials-file option to change the filename in which the client credentials are stored
+- scripts/register_feed2toot_app: --user-credentials-file option to change the filename in which the client credentials are stored
+- scripts/register_feed2toot_app: --name to change the Mastodon app name
+
+## [0.15] - 2020-12-07
+### Changed
+- maintenance version
+- fix bug while using uri_list parameter alone following feedparser upgrade
+- update copyrigth
+- update author email
+
+
+## [0.14] - 2019-12-31
+### Added
+- support for line breaks in toots
+- new configuration parameter toot_max_len to define the maximum length of a toot
+- new configuration parameter no_tags_in_toot to stop hash tags to be added in toots
+
+## [0.13] - 2019-12-27
+### Added
+- new cli option --ignore-ssl and parameter ignore_ssl to ignore ssl error in the rss feeds
+
+### Changed
+- delete the lock file when using --rss-sections cli option
+
+## [0.12] - 2019-08-27
+### Changed
+- simplify code, mostly in the main function
+- fix a bug when title_pattern is used and *_pattern_case_sensitive is not defined
+
+
+## [0.11] - 2019-08-24
+### Added
+- MAJOR CHANGE: command line options --lockfile to define a lock file
+- MAJOR CHANGE: command line options --lock-timeout  to remove this lock file automatically
+- MAJOR CHANGE: lock section in configuration with lock_file andd lock_time parameters
+
+## [0.10] - 2018-09-22
+### Added
+- new syntax for the toot parameter of [rss] section. Use {summary:.100} to cut the rss field summary after the first 100 characters. Contributed by Matthias Henze
+- add the addtags parameter of the [rss] section. Contributed by Matthias Henze
+
+## [0.9] - 2018-06-07
+### Added
+- remove html characters from toots. Contributed by Matthew Lorentz and Simounet
+
+## [0.8] - 2018-05-30
+### Added
+- add the custom parameter in [media] section to join a custom media with every toots
+
+## [0.7] - 2017-09-26
+### Changed
+- fix issue while using uri_list
+
+## [0.6] - 2017-08-02
+### Added
+- define a name for a feed, accessible with {feedname}, contributed by Alexis Metaireau
+- switch the toot visibility. Contributed by The Dod
+- new accept_bozo_exceptions option to allow malformed rss feeds, contributed by Alexis Metaireau
+
+### Changed
+- configuration parser was split into much smaller chunks
+- remove useless imports and coding style, contributed by Alexis Metaireau
+- rephrasing and reformatting of the script register_feed2toot_app, contributed by Bastien Guerry
+
+## [0.5] - 2017-05-05
+### Added
+- manage rss feeds entries without id
+
+## [0.4] - 2017-04-28
+### Changed
+- [hashtaglist] section is not mandatory any more
+- bugfixes
+
 ## [0.3] - 2017-04-12
 ### Changed
 - remove persistentlist dependency because unstable and use own cache storage

Dockerfile

@@ -0,0 +1,13 @@
+FROM python:3.7-bullseye
+
+ADD . /app
+WORKDIR /app
+
+RUN python3 setup.py install; \
+    chmod +x docker_entrypoint.sh; \
+    mkdir -p /data /root/.config; \
+    rm -rf docke.yml
+
+WORKDIR /data
+
+CMD ["/app/docker_entrypoint.sh"]

README.md

@@ -1,63 +1,46 @@
+# Feed2Toot Docker
+
+Feed2Toot Docker is a fork of [chaica/feed2toot](https://gitlab.com/chaica/feed2toot) that allows for simple deployment over Docker. In the future I'll try to add other useful features.
+
+### Usage
+
+With Docker Compose:
+```yaml
+version: '3'
+
+services:
+  feed2toot:
+    image: gitea.massivebox.net/massivebox/feed2toot-docker:latest
+    environment:
+      - USERNAME=sampleusername@sampleinstance.url
+      - PASSWORD=samplepassword
+      - INSTANCE=https://sampleinstance.url
+    restart: always
+    volumes:
+      - ./data:/data
+```
+Insert your full username (including the part with your homeserver's address), password and instance URL where required.
+You also need to create a working `feed2toot.ini` in the `./data` folder. See the [upstream docs](`https://feed2toot.readthedocs.io/en/latest/configure.html#create-feed2toot-configuration`) for the guide.
+
+After the first successful run, you can remove the entire `environment` block if you want, however, if you delete and re-create the container, you will need to put it back.
+
+### Support
+
+Don't ask the upstream developers for help, if something is broken it's more likely that it is my fault.
+You can get support by opening an [issue](https://gitea.massivebox.net/massivebox/feed2toot-docker/issues), or by [contacting me](https://massivebox.net/contact.html), or in the [Matrix support room](https://matrix.to/#/#support:massivebox.net).
+
 ### Feed2toot
 
-Feed2toot automatically parses rss feeds, identifies new posts and posts them on the [Mastodon](https://mastodon.social) social network.
-For the full documentation, [read it online](https://feed2toot.readthedocs.org/en/latest/).
+The following links and addresses are the upstream developers'. I don't want donations for this project, but I encourage you to donate to them.
 
-If you would like, you can [support the development of this project on Liberapay](https://liberapay.com/carlchenet/).
-Alternatively you can donate cryptocurrencies:
-
-- BTC: 1BcdXCcLKN9PRpp6qw23FYkYuVp59dKZix
-- XMR: 4Cxwvw9V6yUehv832FWPTF7FSVuWjuBarFd17QP163uxMaFyoqwmDf1aiRtS5jWgCiRsi73yqedNJJ6V1La2joznUDzkmvBr6KKHT7Dvzj
-
-### Quick Install
-
-* Install Feed2toot from PyPI
-
-        # pip3 install feed2toot
-
-* Install Feed2toot from sources
-  *(see the installation guide for full details)
-  [Installation Guide](http://feed2toot.readthedocs.org/en/latest/install.html)*
-
-
-        # tar zxvf feed2toot-0.3.tar.gz
-        # cd feed2toot
-        # python3 setup.py install
-        # # or
-        # python3 setup.py install --install-scripts=/usr/bin
-
-### Create the authorization for the Feed2toot app
-
-* Just launch the following command::
-
-        $ register_feed2toot_app
-
-### Use Feed2toot
-
-* Create or modify feed2toot.ini file in order to configure feed2toot:
-
-        [mastodon]
-        instance_url=https://mastodon.social
-        user_credentials=feed2toot_usercred.txt
-        client_credentials=feed2toot_clientcred.txt
-
-        [cache]
-        cachefile=cache.db
-
-        [rss]
-        uri=https://www.journalduhacker.net/rss
-        tweet={title} {link}
-
-        [hashtaglist]
-        several_words_hashtags_list=hashtags.txt
-
-* Launch Feed2toot
-
-        $ feed2toot -c /path/to/feed2toot.ini
+- [Liberapay](https://liberapay.com/carlchenet/)
+- BTC: 1AW12Zw93rx4NzWn5evcG7RNNEM2RSLmAC
+- XMR: 43GGv8KzVhxehv832FWPTF7FSVuWjuBarFd17QP163uxMaFyoqwmDf1aiRtS5jWgCiRsi73yqedNJJ6V1La2joznKHGAhDi
 
 ### Authors
 
-* Carl Chenet <chaica@ohmytux.com>
+* MassiveBox <massivebox@massivebox.net>
+* Carl Chenet <carl.chenet@ohmytux.com>
 * Antoine Beaupré <anarcat@debian.org>
 * First developed by Todd Eddy
 

docker_entrypoint.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+cd /data
+
+if [[ ! -f "feed2toot_clientcred.txt" || ! -f "feed2toot_usercred.txt" ]]; then
+    register_feed2toot_app --instance $INSTANCE --username $USERNAME --password $PASSWORD
+fi
+
+while :; do
+    feed2toot -c ./feed2toot.ini
+    sleep 5m
+done

docs/source/authors.rst

@@ -1,4 +1,4 @@
 Authors
 =======
 
-Carl Chenet <chaica@ohmytux.com>
+Carl Chenet <carl.chenet@ohmytux.com>

docs/source/conf.py

@@ -31,6 +31,7 @@ import os
 # ones.
 extensions = [
     'sphinx.ext.autodoc',
+    'sphinx.ext.autosectionlabel'
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -47,16 +48,16 @@ master_doc = 'index'
 
 # General information about the project.
 project = 'feed2toot'
-copyright = '2017, Carl Chenet <chaica@ohmytux.com>'
+copyright = '2015-2021, Carl Chenet <carl.chenet@ohmytux.com>'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
-version = '1.0'
+version = '0.17'
 # The full version, including alpha/beta/rc tags.
-release = '1.0'
+release = '0.17'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -200,7 +201,7 @@ latex_elements = {
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
   ('index', 'feed2toot.tex', 'feed2toot Documentation',
-   'Carl Chenet \\textless{}chaica@ohmytux.com.org\\textgreater{}', 'manual'),
+   'Carl Chenet \\textless{}carl.chenet@ohmytux.com.org\\textgreater{}', 'manual'),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
@@ -230,7 +231,7 @@ latex_documents = [
 # (source start file, name, description, authors, manual section).
 man_pages = [
     ('index', 'feed2toot', 'feed2toot Documentation',
-     ['Carl Chenet <chaica@ohmytux.com>'], 1)
+     ['Carl Chenet <carl.chenet@ohmytux.com>'], 1)
 ]
 
 # If true, show URL addresses after external links.
@@ -244,7 +245,7 @@ man_pages = [
 #  dir menu entry, description, category)
 texinfo_documents = [
   ('index', 'feed2toot', 'feed2toot Documentation',
-   'Carl Chenet <chaica@ohmytux.com>', 'feed2toot', 'One line description of project.',
+   'Carl Chenet <carl.chenet@ohmytux.com>', 'feed2toot', 'One line description of project.',
    'Miscellaneous'),
 ]
 

docs/source/configure.rst

@@ -1,23 +1,34 @@
 Configure Feed2toot
 ===================
 
+Create credentials for Mastodon
+-------------------------------
 As a prerequisite to use Feed2toot, you need to authorize a Mastodon app for your account.
 
 Just use the script register_feed2toot_app to register the feed2toot app for your account.::
 
-    $ ./register_feed2toot_app 
-
-    This app generates Mastodon app credentials needed by Feed2toot.
-    feed2toot_clientcred.txt and feed2toot_usercred.txt will be written in the current dir /home/chaica/progra/python/feed2toot.
-    One connection is initiated to create the app.
+    $ ./register_feed2toot_app
+    
+    This script generates the Mastodon application credentials for Feed2toot.
+    feed2toot_clientcred.txt and feed2toot_usercred.txt will be written
+    in the current directory: /home/me/feed2toot.
+    WARNING: previous files with the same names will be overwritten.
+    
+    A connection is also initiated to create the application.
     Your password is *not* stored.
+    
+    Mastodon instance URL (defaults to https://mastodon.social): https://framapiaf.org
+    Mastodon login: toto@titi.com
+    Mastodon password: 
+    
+    The app feed2toot was added to your Preferences=>Accounts=>Authorized apps page.
+    The file feed2toot_clientcred.txt and feed2toot_usercred.txt were created in the current directory.
 
-    Mastodon instance url (defaults to https://mastodon.social):
-    Mastodon login:chaica@ohmytux.com
-    Mastodon password:
-    The feed2toot app was added to your preferences=>authorized apps page
+As described above, two files were created. See the :ref:`Use register_feed2toot_app` section for more options for register_feed2toot_app.
 
-As described above, two files were created. You'll need them in the feed2toot configuration.
+Create Feed2toot configuration
+------------------------------
+After using register_feed2toot_app, you'll need the credentials in the feed2toot configuration.
 
 In order to configure Feed2toot, you need to create a feed2toot.ini file (or any name you prefer, finishing with the extension .ini) with the following parameters::
 
@@ -26,45 +37,92 @@ In order to configure Feed2toot, you need to create a feed2toot.ini file (or any
     ; Here you need the two files created by register_feed2toot_app
     user_credentials=/etc/feed2toot/credentials/feed2toot_usercred.txt
     client_credentials=/etc/feed2toot/credentials/feed2toot_clientcred.txt
+    ; Default visibility is public, but you can override it:
+    ; toot_visibility=unlisted
 
     [cache]
     cachefile=/var/lib/feed2toot/feed2toot.db
     cache_limit=10000
 
+    [lock]
+    lock_file=/var/lock/feed2toot.lock
+    lock_timeout=3600
+
     [rss]
-    uri: https://www.journalduhacker.net/rss
-    uri_list: /etc/feed2toot//rsslist.txt
-    tweet: {title} {link}
-    title_pattern: Open Source
-    title_pattern_case_sensitive: true
+    uri=https://www.journalduhacker.net/rss
+    uri_list=/etc/feed2toot//rsslist.txt
+    toot={title} {link}
+    ; toot_max_len=125
+    title_pattern=Open Source
+    title_pattern_case_sensitive=true
     no_uri_pattern_no_global_pattern=true
+    ; ignore_ssl=false
 
     [hashtaglist]
-    several_words_hashtags_list: /etc/feed2toot/hashtags.txt
+    several_words_hashtags_list=/etc/feed2toot/hashtags.txt
+    ; no_tags_in_toot=false
+
+    [feedparser]
+    accept_bozo_exceptions=true
+
+    [media]
+    custom=/var/lib/feed2toot/media/logo.png
 
 For the [mastodon] section:
 
 - instance_url: the url of your Mastodon instance
 - user_credentials: a file with the user credentials, generated by the command register_feed2toot_app
 - client_credentials: a file with the client credentials, generated by the command register_feed2toot_app
+- toot_visibility: any of the valid options for the *visibility* field
+  `here`__.
+  Default is *public*, but *unlisted* prevents flooding
+  the instance's public timeline (which is more polite).
+
+__ https://github.com/tootsuite/documentation/blob/master/Using-the-API/API.md#posting-a-new-status
 
 For the [cache] section:
 
-- cachefile: the path to the cache file storing ids of already tweeted links. Absolute path is mandatory. This file should always use the .db extension.
+- cachefile: the path to the cache file storing ids of already tooted links. Absolute path is mandatory. This file should always use the .db extension.
 - cache_limit: length of the cache queue. defaults to 100.
 
+For the [lock] section (starting from version 0.11):
+
+- lock_file: lock to stop any other feed2toot instance to run at the same time. Default is ~/.config/feed2toot.lock
+- lock_timeout: automatically remove the lock if the datetime in the lock file is greater than n seconds. Default is 3600 seconds.
+
 For the [rss] section:
 
 - uri: the url of the rss feed to parse
 - uri_list: a path to a file with several adresses of rss feeds, one by line. Absolute path is mandatory.
-- tweet: format of the tweet you want to post. It should use existing entries of the RSS fields like {title} or {link}. Launch it with this field empty to display all available entries.
+- toot: format of the toot you want to post. It should use existing entries of the RSS fields like {title} or {link}. Launch it with this field empty to display all available entries. If you want to shorten the size of a field, you can use the syntax {summary:.100} to cut the field "summary" of the rss feed after the first 100 characters (starting from version 0.10). To add new lines you can use \n (starting from version 0.14)
+- toot_max_len: the max length of a toot can be defined here. If the toot size is longer, the toot is truncated and "..." added at the end. Defaults is 500 characters.
 - {one field of the rss feed}_pattern: takes a string representing a pattern to match for a specified field of each rss entry of the rss feed, like title_pattern or summary_pattern.
 - {one field of the rss feed}_pattern_case_sensitive: either the pattern matching for the specified field should be case sensitive or not. Default to true if not specified.
 - no_uri_pattern_no_global_pattern: don't apply global pattern (see above) when no pattern-by-uri is defined in the uri_list. Allows to get all entries of a rss in the uri_list because no pattern is defined so we match them all. Defaults to false, meaning the global patterns will be tried on every rss in the uri_list NOT HAVING specific patterns and so ONLY entries from the specific uri in the uri_list matching the global patterns will be considered.
+- addtags: add the tags from the rss feed at the end of the toot. Defaults to true.
+- ignore_ssl: when the uri or uri_list contains an https url with an invalid certificate (e.g an expired one), feed2toot will be unable to get rss content. This option allows to bypass the ssl security to catch the rss content. Defaults to false.
 
 For the [hashtaglist] section:
 
-- several_words_hashtags_list: a path to the file containing hashtags in two or more words. Absolute path is mandatory. By default Feed2toot adds a # before every words of a hashtag.
+- several_words_hashtags_list: a path to the file containing hashtags in two or more words. Absolute path is mandatory. By default Feed2toot adds a # before every words of a hashtag. See documentation below for an example of this file.
+- no_tags_in_toot: stop hash tags to be added at the toot. Defaults to false.
+
+for the [feedparser] section:
+
+- accept_bozo_exceptions: If set to true, feed2toot will accept malformed feeds, which are rejected by default.
+
+For the [media] section:
+
+- custom: the path to a media (should be supported by Mastodon) to be posted with every Mastodon post.
+
+Example of the list of hash tags
+================================
+The list of hash tags is a simple text file with one hash tag composed by several words on a single line::
+
+    free software community
+    open-source
+
+Instead of having #free #software #community or #open-source in the resulting toot, you will have #freesoftwarecommunity and #opensource. You only have to identify the hash tags you frequently use in your RSS feeds and put them in this file to have well formatted hash tags in your toots.
 
 List of rss feeds
 =================
@@ -74,7 +132,7 @@ With the parameter **uri_list**, you can define a list of uri to use. Feed2toot
 
     [rss]
     uri_list=/home/john/feed2toot/rsslist.txt
-    tweet={title} {link}
+    toot={title} {link}
 
 Now let's have a look at the =/home/john/feed2toot/rsslist.txt file::
 
@@ -83,12 +141,21 @@ Now let's have a look at the =/home/john/feed2toot/rsslist.txt file::
 
 Each line of this file is a url to a rss feed. Pretty simple.
 
+Display the name of the feed in the toots
+-----------------------------------------
+
+If you want to display the name of the feed in the resulting toot, you can do so by giving it a name with the following syntax::
+
+    Le journal du hacker <https://www.journalduhacker.net/rss/>
+
+Then in the `toot` configuration, you can use the `{feedname}` syntax, which will be replaced by the actual name of the feed.
+
 Match specific patterns of rss feeds in the uri_list files
 ----------------------------------------------------------
 You can use specific pattern matching for uri in the uri_list file to filter some of the rss entries of a rss feed. Lets modify the previous file::
 
-https://www.journalduhacker.net/rss|title|hacker,psql
-https://carlchenet.com/feed|title|gitlab
+    https://www.journalduhacker.net/rss|title|hacker,psql
+    https://carlchenet.com/feed|title|gitlab
 
 Each line of this file starts with an uri, followed by a pipe (|), followed by the name of the available section to parse (see below), again followed by a pipe (|), followed by patterns, each pattern being separated from the other one by a semi-colon (,).
 
@@ -104,15 +171,9 @@ It is possible to get all entries from a rss feed available in the uri_list file
 
 In you rsslist.txt, just don't give anything else than the needed feed url to get all the entries::
 
-https://www.journalduhacker.net/rss|title|hacker,psql
-https://carlchenet.com/feed|title|gitlab
-https://blog.linuxjobs.fr/feed.php?rss
+    https://www.journalduhacker.net/rss|title|hacker,psql
+    https://carlchenet.com/feed|title|gitlab
+    https://blog.linuxjobs.fr/feed.php?rss
 
-The last line of the file above only has the url of a rss feed. All entries from this feed will be tweeted.
+The last line of the file above only has the url of a rss feed. All entries from this feed will be tooted.
 
-How to display available sections of the rss feed
-=================================================
-Feed2toot offers the **--rss-sections** command line option to display the available section of the rss feed and exits::
-
-    $ feed2toot --rss-sections -c feed2toot.ini
-    The following sections are available in this RSS feed: ['title', 'comments', 'authors', 'link', 'author', 'summary', 'links', 'tags', id', 'author_detail', 'published'].

docs/source/install.rst

@@ -25,7 +25,7 @@ Alternatively, Setuptools may be installed to a user-local path::
 
 * Untar the tarball and go to the source directory with the following commands::
 
-    $ tar zxvf feed2toot-0.3.tar.gz
+    $ tar zxvf feed2toot-0.17.tar.gz
     $ cd feed2toot
 
 * Next, to install Feed2toot on your computer, type the following command with the root user::

docs/source/plugins.rst

@@ -4,7 +4,7 @@ Feed2toot supports plugins. Plugins offer optional features, not supported by de
 
 InfluxDB
 --------
-The InfluxDB plugin allows to store already published tweets in a InfluxDB database.
+The InfluxDB plugin allows to store already published toots in a InfluxDB database.
 
 Install the InfluxDB plugin
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -28,7 +28,7 @@ Below is the block of configuration to add in your feed2toot.ini::
     user=influxuser
     pass=V3ryS3cr3t
     database=influxdb
-    measurement=tweets
+    measurement=toots
 
 - host: the host where the influxdb instance is. Defaults to 127.0.0.1
 - port: the port where the influxdb instance is listening to. Defaults to 8086

docs/source/use.rst

@@ -5,8 +5,8 @@ After the configuration of Feed2toot, just launch the following command::
     $ feed2toot -c /path/to/feed2toot.ini
 
 Run Feed2toot on a regular basis
-=================================
-Feed2toot should be launche on a regular basis in order to efficiently send your new RSS entries to Mastodon. It is quite easy to achieve with adding a line to your user crontab, as described below::
+---------------------------------
+Feed2toot should be launched on a regular basis in order to efficiently send your new RSS entries to Mastodon. It is quite easy to achieve by adding a line to your user crontab, as described below::
 
     @hourly feed2toot -c /path/to/feed2toot.ini
 
@@ -15,19 +15,19 @@ will execute feed2toot every hour. Or without the syntactic sugar in the global
     0 * * * * johndoe feed2toot -c /path/to/feed2toot.ini
 
 Test option
-===========
+-----------
 In order to know what's going to be sent to Mastodon without actually doing it, use the **--dry-run** option::
 
     $ feed2toot --dry-run -c /path/to/feed2toot.ini
 
 Debug option
-============
+------------
 In order to increase the verbosity of what's Feed2toot is doing, use the **--debug** option followed by the level of verbosity see [the the available different levels](https://docs.python.org/3/library/logging.html)::
 
     $ feed2toot --debug -c /path/to/feed2toot.ini
 
-Populate the cache file without posting tweets
-==============================================
+Populate the cache file without posting toots
+---------------------------------------------
 Starting from 0.8, Feed2toot offers the **--populate-cache** command line option to populate the cache file without posting to Mastodon::
 
     $ feed2toot --populate-cache -c feed2toot.ini
@@ -43,14 +43,55 @@ Starting from 0.8, Feed2toot offers the **--populate-cache** command line option
     populating RSS entry https://www.journalduhacker.net/s/lqswmz
 
 How to display available sections of the rss feed
-=================================================
+-------------------------------------------------
 Starting from 0.8, Feed2toot offers the **--rss-sections** command line option to display the available section of the rss feed and exits::
 
     $ feed2toot --rss-sections -c feed2toot.ini
     The following sections are available in this RSS feed: ['title', 'comments', 'authors', 'link', 'author', 'summary', 'links', 'tags', id', 'author_detail', 'published'].
 
 Using syslog
-============
+------------
 Feed2toot is able to send its log to syslog. You can use it with the following command::
 
     $ feed2toot --syslog=WARN -c /path/to/feed2toot.ini
+
+Limit number of rss entries published at each execution
+-------------------------------------------------------
+If you want to limit the number of rss entries published at each execution, you can use the --limit CLI option.
+
+    $ feed2toot --limit 5 -c /path/to/feed2toot.ini
+
+The number of posts to Mastodon will be at 5 posts top with this CLI option.
+
+Use register_feed2toot_app
+==========================
+You need a Mastodon app associated to a user on the Mastodon instance. The script register_feed2toot_app will create an app for Feed2toot and upload it on the specified Mastodon instance.
+
+Primary usage ::
+
+    $ register_feed2toot_app
+
+Possible CLI options:
+
+- use the **--client-credentials-file** option to change the filename in which the client credentials are stored (defaults to feed2toot_clientcred.txt)
+- use the **--user-credentials-file** option to change the filename in which the user credentials are stored (defaults to feed2toot_usercred.txt)
+- use the **--name** to change the Mastodon app name (defaults to feed2toot)
+
+Example with full options and full output::
+
+    $ ./register_feed2toot_app --user-credentials-file f2tusercreds.txt --client-credentials-file f2tclientcreds.txt --name f2t
+    
+    This script generates the Mastodon application credentials for Feed2toot.
+    f2tclientcreds.txt and f2tusercreds.txt will be written
+    in the current directory: /home/me/feed2toot/scripts.
+    WARNING: previous files with the same names will be overwritten.
+    
+    A connection is also initiated to create the application.
+    Your password is *not* stored.
+    
+    Mastodon instance URL (defaults to https://mastodon.social): https://framapiaf.org
+    Mastodon login: toto@titi.com
+    Mastodon password: 
+
+    The app f2t was added to your preferences=>authorized apps page.
+    The file f2tclientcreds.txt and f2tusercreds.txt were created in the current directory.

feed2toot.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # vim:ts=4:sw=4:ft=python:fileencoding=utf-8
-# Copyright © 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or

feed2toot/__init__.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # vim:ts=4:sw=4:ft=python:fileencoding=utf-8
-# Copyright © 2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or

feed2toot/addtags.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # vim:ts=4:sw=4:ft=python:fileencoding=utf-8
-# Copyright © 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
@@ -20,7 +20,7 @@
 # standard library imports
 from operator import itemgetter
 
-class AddTags(object):
+class AddTags:
     '''Add as many tags as possible depending on the tweet length'''
     def __init__(self, tweet, tags):
         '''Constructor of AddTags class'''
@@ -31,18 +31,17 @@ class AddTags(object):
     def main(self):
         '''Main of the AddTags class class'''
         maxlength = 500
-        shortenedlinklength = 23
         tweetlength = len(self.tweet)
 
         # sort list of tags, the ones with the greater length first
-        tagswithindices = ({'text':i, 'length':len(i)} for i in self.tags)
+        tagswithindices = ({'text':i, 'length': len(i)} for i in self.tags)
         sortedtagswithindices = sorted(tagswithindices, key=itemgetter('length'), reverse=True)
         self.tags = (i['text'] for i in sortedtagswithindices)
 
         # add tags is space is available
         for tag in self.tags:
             taglength = len(tag)
-            if (tweetlength + (taglength +1)) <= maxlength:
+            if (tweetlength + (taglength + 1)) <= maxlength:
                 self.tweet = ' '.join([self.tweet, tag])
                 tweetlength += (taglength + 1)
 

feed2toot/cliparse.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright © 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
@@ -20,12 +20,13 @@
 from argparse import ArgumentParser
 import glob
 import logging
+import os
 import os.path
 import sys
 
-__version__ = '0.3'
+__version__ = '0.17'
 
-class CliParse(object):
+class CliParse:
     '''CliParse class'''
     def __init__(self):
         '''Constructor for the CliParse class'''
@@ -33,8 +34,8 @@ class CliParse(object):
 
     def main(self):
         '''main of CliParse class'''
-        feed2tootepilog = 'For more information: https://feed2toot.readhthedocs.org'
-        feed2tootdescription = 'Take rss feed and send it to Mastodon' 
+        feed2tootepilog = 'For more information: https://feed2toot.readthedocs.io'
+        feed2tootdescription = 'Take rss feed and send it to Mastodon'
         parser = ArgumentParser(prog='feed2toot',
                                 description=feed2tootdescription,
                                 epilog=feed2tootepilog)
@@ -49,10 +50,19 @@ class CliParse(object):
         parser.add_argument('-a', '--all', action='store_true', default=False,
                             dest='all',
                             help='tweet all RSS items, regardless of cache')
+        parser.add_argument('--ignore-ssl', action='store_true', default=False,
+                            dest='ignore_ssl',
+                            help='ignore ssl errors while fetching rss feeds')
         parser.add_argument('-l', '--limit', dest='limit', default=10, type=int,
                             help='tweet only LIMIT items (default: %(default)s)')
+        parser.add_argument('-t', '--lock-timeout', dest='locktimeout', default=3600, type=int,
+                            help='lock timeout in seconds after which feed2toot can removes the lock itself')
         parser.add_argument('--cachefile', dest='cachefile',
                             help='location of the cache file (default: %(default)s)')
+        parser.add_argument('--lockfile', dest='lockfile',
+                            default=os.path.join(os.getenv('XDG_CONFIG_HOME', '~/.config'),
+                                                  'feed2toot.lock'),
+                            help='location of the lock file (default: %(default)s)')
         parser.add_argument('-n', '--dry-run', dest='dryrun',
                             action='store_true', default=False,
                             help='Do not actually post tweets')
@@ -63,7 +73,7 @@ class CliParse(object):
                             action='store_const', const='debug', default='warning',
                             help='enable debug output, work on log level DEBUG')
         levels = [i for i in logging._nameToLevel.keys()
-        if (type(i) == str and i != 'NOTSET')]
+                  if (type(i) == str and i != 'NOTSET')]
         parser.add_argument('--syslog', nargs='?', default=None,
                             type=str.upper, action='store',
                             const='INFO', choices=levels,
@@ -71,7 +81,7 @@ class CliParse(object):
                             logging, INFO if --syslog is specified without
                             argument""")
         parser.add_argument('--hashtaglist', dest='hashtaglist',
-                            help='a list of hashtag to match')
+                            help='a list of hashtags to match')
         parser.add_argument('-p', '--populate-cache', action='store_true', default=False,
                             dest='populate',
                             help='populate RSS entries in cache without actually posting them to Mastodon')
@@ -98,6 +108,11 @@ class CliParse(object):
         # verify if a configuration file is provided
         if not self.opts.configs:
             sys.exit('no configuration file was found at the specified path(s) with the option -c')
+        # verify the path to the hashtaglist
+        if self.opts.hashtaglist:
+            hashtaglist = os.path.expanduser(self.opts.hashtaglist)
+            if not os.path.exists(hashtaglist):
+                sys.exit('the {hashtaglist} file does not seem to exist, please provide a valid path'.format(hashtaglist=hashtaglist))
 
     @property
     def options(self):

feed2toot/confparse.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright © 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
@@ -17,17 +17,33 @@
 '''Get values of the configuration file'''
 
 # standard library imports
-from configparser import SafeConfigParser, NoOptionError, NoSectionError
+from configparser import SafeConfigParser
 import logging
 import os
 import os.path
-import socket
 import sys
+import re
 
 # 3rd party library imports
 import feedparser
 
-class ConfParse(object):
+# feed2toot library imports
+from feed2toot.confparsers.cache import parsecache
+from feed2toot.confparsers.hashtaglist import parsehashtaglist
+from feed2toot.confparsers.hashtags.nohashtags import parsenotagsintoot
+from feed2toot.confparsers.feedparser import parsefeedparser
+from feed2toot.confparsers.lock import parselock
+from feed2toot.confparsers.media import parsemedia
+from feed2toot.confparsers.plugins import parseplugins
+from feed2toot.confparsers.rss.ignoressl import parseignoressl
+from feed2toot.confparsers.rss.pattern import parsepattern
+from feed2toot.confparsers.rss.toot import parsetoot
+from feed2toot.confparsers.rss.uri import parseuri
+from feed2toot.confparsers.rss.urilist import parseurilist
+from feed2toot.confparsers.rss.addtags import parseaddtags
+from feed2toot.confparsers.rss.tootmaxlen import parsetootmaxlen
+
+class ConfParse:
     '''ConfParse class'''
     def __init__(self, clioptions):
         '''Constructor of the ConfParse class'''
@@ -45,186 +61,69 @@ class ConfParse(object):
             config = SafeConfigParser()
             if not config.read(os.path.expanduser(pathtoconfig)):
                 sys.exit('Could not read config file')
+            ####################
+            # feedparser section
+            ####################
+            accept_bozo_exceptions = parsefeedparser(config)
             ###########################
-            # 
             # the rss section
-            # 
             ###########################
-            section = 'rss'
-            if config.has_section(section):
-                ############################
-                # tweet option
-                ############################
-                confoption = 'tweet'
-                if config.has_option(section, confoption):
-                    self.tweetformat = config.get(section, confoption)
-                else:
-                    sys.exit('You should define a format for your tweet with the keyword "tweet" in the [rss] section')
-                ############################
-                # pattern format option
-                ############################
-                options['patterns'] = {}
-                options['patternscasesensitive'] = {}
-                for pattern in ['summary_detail', 'published_parsed', 'guidislink', 'authors', 'links', 'title_detail', 'author', 'author_detail', 'comments', 'published', 'summary', 'tags', 'title', 'link', 'id']:
-                    currentoption = '{}_pattern'.format(pattern)
-                    if config.has_option(section, currentoption):
-                        tmppattern = config.get(section, currentoption)
-                        if self.stringsep in tmppattern:
-                            options['patterns'][currentoption] = [i for i in tmppattern.split(self.stringsep) if i]
-                        else:
-                            options['patterns'][currentoption] = [tmppattern]
-
-                    # pattern_case_sensitive format
-                    currentoption = '{}_pattern_case_sensitive'.format(pattern)
-                    if config.has_option(section, currentoption):
-                        try:
-                            options['patternscasesensitive'][currentoption] = config.getboolean(section, currentoption)
-                        except ValueError as err:
-                            print(err)
-                            options['patternscasesensitive'][currentoption] = True
-                ############################
-                # rsslist
-                ############################
-                bozoexception = False
-                feeds = []
-                patterns = []
-                currentoption = 'uri_list'
-                if config.has_option(section, currentoption):
-                    rssfile = config.get(section, currentoption)
-                    rssfile = os.path.expanduser(rssfile)
-                    if not os.path.exists(rssfile) or not os.path.isfile(rssfile):
-                        sys.exit('The path to the uri_list parameter is not valid: {rssfile}'.format(rssfile=rssfile))
-                    rsslist = open(rssfile, 'r').readlines()
-                    for line in rsslist:
-                        line = line.strip()
-                        # split each line in two parts, rss link and a string with the different patterns to look for
-                        confobjects = line.split('|')
-                        if len(confobjects) > 3 or len(confobjects) == 2:
-                            sys.exit('This line in the list of uri to parse is not formatted correctly: {line}'.format(line))
-                        if len(confobjects) == 3:
-                            rss, rssobject, patternstring = line.split('|')
-                        if len(confobjects) == 1:
-                            rss = confobjects[0]
-                            rssobject = ''
-                            patternstring = ''
-                        # split different searched patterns
-                        patterns = [i for i in patternstring.split(self.stringsep) if i]
-                        # retrieve the content of the rss
-                        feed = feedparser.parse(rss)
-                        if 'bozo_exception' in feed:
-                            bozoexception = True
-                            logging.warning(feed['bozo_exception'])
-                            continue
-                        # check if the rss feed and the rss entry are valid ones
-                        if 'entries' in feed:
-                            if rssobject and rssobject not in feed['entries'][0].keys():
-                                sys.exit('The rss object {rssobject} could not be found in the feed {rss}'.format(rssobject=rssobject, rss=rss))
-                        else:
-                            sys.exit('The rss feed {rss} does not seem to be valid'.format(rss=rss))
-                        feeds.append({'feed': feed, 'patterns': patterns, 'rssobject': rssobject})
-                    # test if all feeds in the list were unsuccessfully retrieved and if so, leave
-                    if not feeds and bozoexception:
-                        sys.exit('No feed could be retrieved. Leaving.')
-                ############################
-                # uri
-                ############################
-                if not feeds and not self.clioptions.rss_uri:
-                    confoption = 'uri'
-                    if config.has_option(section, confoption):
-                        options['rss_uri'] = config.get('rss', 'uri')
-                    else:
-                        sys.exit('{confoption} parameter in the [{section}] section of the configuration file is mandatory. Exiting.'.format(section=section, confoption=confoption))
-                else:
-                    options['rss_uri'] = self.clioptions.rss_uri
-                # get the rss feed for rss parameter of [rss] section
-                feed = feedparser.parse(options['rss_uri'])
-                if not feed:
-                    sys.exit('Unable to parse the feed at the following url: {rss}'.format(rss=rss))
-
-                #########################################
-                # no_uri_pattern_no_global_pattern option
-                #########################################
-                currentoption = 'no_uri_pattern_no_global_pattern'
-                # default value
-                options['nopatternurinoglobalpattern'] = False
-                if config.has_option(section, currentoption):
-                    options['nopatternurinoglobalpattern'] = config.getboolean(section, currentoption)
+            self.tweetformat = parsetoot(config)
+            options['tootmaxlen'] = parsetootmaxlen(config)
+            #################################################
+            # pattern and patter_case_sensitive format option
+            #################################################
+            options['patterns'], options['patternscasesensitive'] = parsepattern(config)
+            #################################################
+            # lock file options
+            #################################################
+            options['lockfile'], options['locktimeout'] = parselock(self.clioptions.lockfile, self.clioptions.locktimeout, config)
+            ###############################
+            # addtags option, default: True
+            ###############################
+            options['addtags'] = parseaddtags(config)
+            ###################
+            # ignore_ssl option
+            ###################
+            ignore_ssl = parseignoressl(config, self.clioptions.ignore_ssl)
+            #################
+            # uri_list option
+            #################
+            feeds = []
+            feeds = parseurilist(config, accept_bozo_exceptions, ignore_ssl)
+            ############
+            # uri option
+            ############
+            if config.has_option('rss', 'uri') or self.clioptions.rss_uri:
+                options['rss_uri'], feed, feedname, options['nopatternurinoglobalpattern'] = parseuri(config, self.clioptions.rss_uri, feeds, ignore_ssl)
+            else:
+                if config.has_option('rss', 'no_uri_pattern_no_global_pattern'):
+                    options['nopatternurinoglobalpattern'] = config.getboolean('rss', 'no_uri_pattern_no_global_pattern')
             ###########################
-            # 
             # the cache section
-            # 
             ###########################
-            section = 'cache'
-            if not self.clioptions.cachefile:
-                confoption = 'cachefile'
-                if config.has_section(section):
-                    options['cachefile'] = config.get(section, confoption)
-                else:
-                    sys.exit('You should provide a {confoption} parameter in the [{section}] section'.format(section=section, confoption=confoption))
-                options['cachefile'] = os.path.expanduser(options['cachefile'])
-                cachefileparent = os.path.dirname(options['cachefile'])
-                if cachefileparent and not os.path.exists(cachefileparent):
-                    sys.exit('The parent directory of the cache file does not exist: {cachefileparent}'.format(cachefileparent=cachefileparent))
-            else:
-                options['cachefile'] = self.clioptions.cachefile
-            ### cache limit
-            if config.has_section(section):
-                confoption = 'cache_limit'
-                if config.has_option(section, confoption):
-                    try:
-                        options['cache_limit'] = int(config.get(section, confoption))
-                    except ValueError as err:
-                        sys.exit('Error in configuration with the {confoption} parameter in [{section}]: {err}'.format(confoption=confoption, section=section, err=err))
-                else:
-                    options['cache_limit'] = 100
-            else:
-                options['cache_limit'] = 100
+            options['cachefile'], options['cache_limit'] = parsecache(self.clioptions.cachefile, config)
             ###########################
-            # 
-            # the hashtag section
-            # 
+            # the hashtaglist section
             ###########################
-            section = 'hashtaglist'
-            if not self.clioptions.hashtaglist:
-                confoption = 'several_words_hashtags_list'
-                if config.has_section(section):
-                    options['hashtaglist'] = config.get(section, confoption)
-                    options['hashtaglist'] = os.path.expanduser(options['hashtaglist'])
-                    if not os.path.exists(options['hashtaglist']) or not os.path.isfile(options['hashtaglist']):
-                        sys.exit('The path to the several_words_hashtags_list parameter is not valid: {hashtaglist}'.format(hashtaglist=options['hashtaglist']))
-                else:
-                    options['hashtaglist'] = False
+            options['hashtaglist'] = parsehashtaglist(self.clioptions.hashtaglist, config)
+            options['notagsintoot'] = parsenotagsintoot(config)
+            ###########################
+            # the media section
+            ###########################
+            options['media'] = parsemedia(config)
             ###########################
-            # 
             # the plugins section
-            # 
             ###########################
-            plugins = {}
-            section = 'influxdb'
-            if config.has_section(section):
-                ##########################################
-                # host, port, user, pass, database options
-                ##########################################
-                plugins[section] = {}
-                for currentoption in ['host','port','user','pass','database']:
-                    if config.has_option(section, currentoption):
-                        plugins[section][currentoption] = config.get(section, currentoption)
-                if 'host' not in plugins[section]:
-                    plugins[section]['host'] = '127.0.0.1'
-                if 'port' not in plugins[section]:
-                    plugins[section]['port'] = 8086
-                if 'measurement' not in plugins[section]:
-                    plugins[section]['measurement'] = 'tweets'
-                for field in ['user','pass','database']:
-                    if field not in plugins[section]:
-                        sys.exit('Parsing error for {field} in the [{section}] section: {field} is not defined'.format(field=field, section=section))
-
-            # create the returned object with previously parsed data
+            plugins = parseplugins(config)
+            ########################################
+            # return the final configurations values
+            ########################################
             if feeds:
                 self.confs.append((options, config, self.tweetformat, feeds, plugins))
             else:
-                self.confs.append((options, config, self.tweetformat, [{'feed': feed, 'patterns': [], 'rssobject': ''}], plugins))
-        
+                self.confs.append((options, config, self.tweetformat, [{'feed': feed, 'patterns': [], 'rssobject': '', 'feedname': feedname}], plugins))
+
     @property
     def confvalues(self):
         '''Return the values of the different configuration files'''

feed2toot/confparsers/__init__.py

@@ -0,0 +1,15 @@
+#!/usr/bin/env python3
+# vim:ts=4:sw=4:ft=python:fileencoding=utf-8
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>

feed2toot/confparsers/cache.py

@@ -0,0 +1,55 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get values of the cache section
+'''Get values of the cache section'''
+
+# standard library imports
+import os.path
+import sys
+
+def parsecache(clioption, config):
+    '''Parse configuration values and get values of the hashtaglist section'''
+    cachefile = ''
+    cachelimit = 100
+    section = 'cache'
+    if not clioption:
+        ##################
+        # cachefile option
+        ##################
+        confoption = 'cachefile'
+        if config.has_section(section):
+            cachefile = config.get(section, confoption)
+        else:
+            sys.exit('You should provide a {confoption} parameter in the [{section}] section'.format(section=section, confoption=confoption))
+        cachefile = os.path.expanduser(cachefile)
+        cachefileparent = os.path.dirname(cachefile)
+        if cachefileparent and not os.path.exists(cachefileparent):
+            sys.exit('The parent directory of the cache file does not exist: {cachefileparent}'.format(cachefileparent=cachefileparent))
+    else:
+        cachefile = clioption
+    ####################
+    # cache_limit option
+    ####################
+    if config.has_section(section):
+        confoption = 'cache_limit'
+        if config.has_option(section, confoption):
+            try:
+                cachelimit = int(config.get(section, confoption))
+            except ValueError as err:
+                sys.exit('Error in configuration with the {confoption} parameter in [{section}]: {err}'.format(confoption=confoption, section=section, err=err))
+        else:
+            cachelimit = 100
+    return cachefile, cachelimit

feed2toot/confparsers/feedparser.py

@@ -0,0 +1,29 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get values of the feedparser section
+'''Get values of the feedparser section'''
+
+# standard library imports
+import sys
+
+def parsefeedparser(config):
+    '''Parse configuration values and get values of the feedparser section'''
+    section = 'feedparser'
+    option = 'accept_bozo_exceptions'
+    accept_bozo_exceptions = False
+    if config.has_option(section, option):
+        accept_bozo_exceptions = config.getboolean(section, option)
+    return accept_bozo_exceptions

feed2toot/confparsers/hashtaglist.py

@@ -0,0 +1,38 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get values of the hashtaglist section
+'''Get values of the hashtaglist section'''
+
+# standard library imports
+import os.path
+import sys
+
+def parsehashtaglist(clioption, config):
+    '''Parse configuration values and get values of the hashtaglist section'''
+    hashtaglist = ''
+    section = 'hashtaglist'
+    if not clioption:
+        ####################################
+        # several_words_hashtags_list option
+        ####################################
+        confoption = 'several_words_hashtags_list'
+        if config.has_section(section):
+            if config.has_option(section, confoption):
+                hashtaglist = config.get(section, confoption)
+                hashtaglist = os.path.expanduser(hashtaglist)
+                if not os.path.exists(hashtaglist) or not os.path.isfile(hashtaglist):
+                    sys.exit('The path to the several_words_hashtags_list parameter is not valid: {hashtaglist}'.format(hashtaglist=hashtaglist))
+    return hashtaglist

feed2toot/confparsers/hashtags/__init__.py

@@ -0,0 +1,15 @@
+#!/usr/bin/env python3
+# vim:ts=4:sw=4:ft=python:fileencoding=utf-8
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>

feed2toot/confparsers/hashtags/nohashtags.py

@@ -0,0 +1,26 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get values of the no_tags_in_toot option of the hashtaglist section
+'''Get values of the no_tags_in_toot option of the hashtaglist section'''
+
+def parsenotagsintoot(config):
+    '''Parse configuration values and get values of the the no_tags_in_toot option of the hashtaglist section'''
+    section = 'hashtaglist'
+    option = 'no_tags_in_toot'
+    notagsintoot = False
+    if config.has_option(section, option):
+        notagsintoot = config.getboolean(section, option)
+    return notagsintoot

feed2toot/confparsers/lock.py

@@ -0,0 +1,48 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get values of the lock section
+'''Get values of the lock section'''
+
+# standard library imports
+import os.path
+import sys
+
+def parselock(lockfile, locktimeout, config):
+    '''Parse configuration values and get values of the hashtaglist section'''
+    lockfile = lockfile
+    locktimeout = locktimeout
+    section = 'lock'
+    ##################
+    # lockfile option
+    ##################
+    confoption = 'lock_file'
+    if config.has_section(section):
+        lockfile = config.get(section, confoption)
+    lockfile = os.path.expanduser(lockfile)
+    lockfileparent = os.path.dirname(lockfile)
+    if lockfileparent and not os.path.exists(lockfileparent):
+        sys.exit('The parent directory of the lock file does not exist: {lockfileparent}'.format(lockfileparent=lockfileparent))
+    ######################
+    # lock_timeout option
+    ######################
+    if config.has_section(section):
+        confoption = 'lock_timeout'
+        if config.has_option(section, confoption):
+            try:
+                locktimeout = int(config.get(section, confoption))
+            except ValueError as err:
+                sys.exit('Error in configuration with the {confoption} parameter in [{section}]: {err}'.format(confoption=confoption, section=section, err=err))
+    return lockfile, locktimeout

feed2toot/confparsers/media.py

@@ -0,0 +1,39 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get values of the media section
+'''Get values of the media section'''
+
+# standard library imports
+import os.path
+import sys
+
+def parsemedia(config):
+    '''Parse configuration values and get values of the media section'''
+    mediaconf = {}
+    section = 'media'
+    ####################################
+    # media option
+    ####################################
+    confoption = 'custom'
+    if config.has_section(section):
+        if config.has_option(section, confoption):
+            media = config.get(section, confoption)
+            media = os.path.expanduser(media)
+            if not os.path.exists(media) or not os.path.isfile(media):
+                sys.exit('The path to the custom parameter is not valid: {media}'.format(media=media))
+            else:
+                mediaconf[confoption] = media
+    return mediaconf

feed2toot/confparsers/plugins.py

@@ -0,0 +1,43 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get values of the plugins section
+'''Get values of the plugins section'''
+
+# standard library imports
+import sys
+
+def parseplugins(config):
+    '''Parse configuration values and get values of the plugins section'''
+    plugins = {}
+    section = 'influxdb'
+    if config.has_section(section):
+        ##########################################
+        # host, port, user, pass, database options
+        ##########################################
+        plugins[section] = {}
+        for currentoption in ['host', 'port', 'user', 'pass', 'database', 'measurement']:
+            if config.has_option(section, currentoption):
+                plugins[section][currentoption] = config.get(section, currentoption)
+        if 'host' not in plugins[section]:
+            plugins[section]['host'] = '127.0.0.1'
+        if 'port' not in plugins[section]:
+            plugins[section]['port'] = 8086
+        if 'measurement' not in plugins[section]:
+            plugins[section]['measurement'] = 'toots'
+        for field in ['user', 'pass', 'database']:
+            if field not in plugins[section]:
+                sys.exit('Parsing error for {field} in the [{section}] section: {field} is not defined'.format(field=field, section=section))
+    return plugins

feed2toot/confparsers/rss/__init__.py

@@ -0,0 +1,15 @@
+#!/usr/bin/env python3
+# vim:ts=4:sw=4:ft=python:fileencoding=utf-8
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>

feed2toot/confparsers/rss/addtags.py

@@ -0,0 +1,34 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get value of the patterne option of rss section
+'''Get value of the addtags option of the rss section'''
+
+# standard library imports
+import logging
+import sys
+
+def parseaddtags(config):
+    '''Parse configuration value of the addtags option of the rss section'''
+    addtags = True
+    section = 'rss'
+    if config.has_section(section):
+        if config.has_option(section, 'addtags'):
+            try:
+                addtags = config.getboolean(section, 'addtags')
+            except ValueError as err:
+                logging.warn(err)
+                addtags = True
+    return addtags

feed2toot/confparsers/rss/ignoressl.py

@@ -0,0 +1,30 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get values of the ignoressl option of the rss section
+'''Get values of the ignoressl option of the rss section'''
+
+# standard library imports
+import ssl
+
+def parseignoressl(config, ignore_ssl_from_cli):
+    '''Parse configuration values and get values of the feedparser section'''
+    section = 'rss'
+    option = 'ignore_ssl'
+    if config.has_option(section, option):
+        ignoressl = config.getboolean(section, option)
+    else:
+        ignoressl = ignore_ssl_from_cli
+    return ignoressl

feed2toot/confparsers/rss/pattern.py

@@ -0,0 +1,55 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get value of the patterne option of rss section
+'''Get value of the pattern option of the rss section'''
+
+# standard library imports
+import logging
+import sys
+
+def parsepattern(config):
+    '''Parse configuration value of the pattern option of the rss section'''
+    patterns = {}
+    patternscasesensitive = {}
+    stringsep = ','
+    section = 'rss'
+    if config.has_section(section):
+        #######################
+        # pattern format option
+        #######################
+        for pattern in ['summary_detail', 'published_parsed', 'guidislink', 'authors', 'links', 'title_detail', 'author', 'author_detail', 'comments', 'published', 'summary', 'tags', 'title', 'link', 'id']:
+            currentoption = '{}_pattern'.format(pattern)
+            if config.has_option(section, currentoption):
+                tmppattern = config.get(section, currentoption)
+                if stringsep in tmppattern:
+                    patterns[currentoption] = [i for i in tmppattern.split(stringsep) if i]
+                else:
+                    patterns[currentoption] = [tmppattern]
+
+            ###############################
+            # pattern_case_sensitive option
+            ###############################
+            currentoption = '{}_pattern_case_sensitive'.format(pattern)
+            if config.has_option(section, currentoption):
+                try:
+                    patternscasesensitive[currentoption] = config.getboolean(section, currentoption)
+                except ValueError as err:
+                    logging.warn(err)
+                    patternscasesensitive[currentoption] = True
+            else:
+                # default value
+                patternscasesensitive[currentoption] = False
+    return patterns, patternscasesensitive

feed2toot/confparsers/rss/toot.py

@@ -0,0 +1,40 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get value of the toot/tweet option of rss section
+'''Get value of the toot/tweet option of the rss section'''
+
+# standard library imports
+import sys
+import logging
+
+def parsetoot(config):
+    '''Parse configuration value of the toot/tweet optionof the rss section'''
+    section = 'rss'
+    if config.has_section(section):
+        ############################
+        # tweet option
+        ############################
+        oldconfoption = 'tweet'
+        confoption = 'toot'
+        # manage 'tweet' for compatibility reason with first versions
+        if config.has_option(section, oldconfoption):
+            logging.warn("Your configuration file uses a 'tweet' parameter instead of 'toot'. 'tweet' is deprecated and will be removed in Feed2toot 0.7")
+            tootformat = config.get(section, oldconfoption)
+        elif config.has_option(section, confoption):
+            tootformat = config.get(section, confoption)
+        else:
+            sys.exit('You should define a format for your tweet with the parameter "{confoption}" in the [{section}] section'.format(confoption=confoption, section=section))
+    return tootformat

feed2toot/confparsers/rss/tootmaxlen.py

@@ -0,0 +1,37 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get value of the toot/tweet option of rss section
+'''Get value of the toot/tweet option of the rss section'''
+
+# standard library imports
+import sys
+import logging
+
+def parsetootmaxlen(config):
+    '''Parse configuration value of the toot_max_len option of the rss section'''
+    section = 'rss'
+    tootmaxlen = 500
+    if config.has_section(section):
+        ############################
+        # toot_max_len parameter
+        ############################
+        confoption = 'toot_max_len'
+        if config.has_option(section, confoption):
+            try:
+                tootmaxlen = int(config.get(section, confoption))
+            except ValueError as err:
+                sys.exit('Error in configuration with the {confoption} parameter in [{section}]: {err}'.format(confoption=confoption, section=section, err=err))
+    return tootmaxlen

feed2toot/confparsers/rss/uri.py

@@ -0,0 +1,67 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get value of the uri option of rss section
+'''Get value of the uri option of the rss section'''
+
+# standard library imports
+import feedparser
+import ssl
+import sys
+import re
+
+def parseuri(config, clioption, feeds, ignoressl):
+    '''Parse configuration value of the uri option of the rss section'''
+    rssuri = ''
+    feedname =''
+    nopatternurinoglobalpattern = False
+    section = 'rss'
+    if config.has_section(section):
+        ############
+        # uri option
+        ############
+        if not feeds and not clioption:
+            confoption = 'uri'
+            if config.has_option(section, confoption):
+                urifeed = config.get('rss', 'uri')
+                feedname = None
+                if '<' in urifeed:
+                    matches = re.match('(.*) <(.*)>', urifeed)
+                    if not matches:
+                        sys.exit('This uri to parse is not formatted correctly: {urifeed}'.format(urifeed))
+                    feedname, finaluri = matches.groups()
+                    rssuri = finaluri
+                else:
+                    rssuri = config.get('rss', 'uri')
+            else:
+                sys.exit('{confoption} parameter in the [{section}] section of the configuration file is mandatory. Exiting.'.format(section=section, confoption=confoption))
+        else:
+            rssuri = clioption
+        # ignore ssl if asked
+        if ignoressl:
+            if hasattr(ssl, '_create_unverified_context'):
+                ssl._create_default_https_context = ssl._create_unverified_context
+        # get the rss feed for rss parameter of [rss] section
+        feed = feedparser.parse(rssuri)
+        if not feed:
+            sys.exit('Unable to parse the feed at the following url: {rss}'.format(rss=rss))
+        #########################################
+        # no_uri_pattern_no_global_pattern option
+        #########################################
+        currentoption = 'no_uri_pattern_no_global_pattern'
+        # default value
+        if config.has_option(section, currentoption):
+            nopatternurinoglobalpattern = config.getboolean(section, currentoption)
+        return rssuri, feed, feedname, nopatternurinoglobalpattern

feed2toot/confparsers/rss/urilist.py

@@ -0,0 +1,87 @@
+# -*- coding: utf-8 -*-
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/
+
+# Get value of the uri_list option of rss section
+'''Get value of the uri_list option of the rss section'''
+
+# standard library imports
+import feedparser
+import logging
+import os.path
+import ssl
+import sys
+import re
+
+def parseurilist(config, accept_bozo_exceptions, ignoressl):
+    '''Parse configuration value of the uri_list option of the rss section'''
+    bozoexception = False
+    feeds = []
+    patterns = []
+    section = 'rss'
+    stringsep = ','
+    if config.has_section(section):
+        #################
+        # uri_list option
+        #################
+        currentoption = 'uri_list'
+        if config.has_option(section, currentoption):
+            rssfile = config.get(section, currentoption)
+            rssfile = os.path.expanduser(rssfile)
+            if not os.path.exists(rssfile) or not os.path.isfile(rssfile):
+                sys.exit('The path to the uri_list parameter is not valid: {rssfile}'.format(rssfile=rssfile))
+            with open(rssfile, 'r') as rsfo:
+                rsslist = rsfo.readlines()
+            for line in rsslist:
+                line = line.strip()
+                # split each line in two parts, rss link and a string with the different patterns to look for
+                feedname = ''
+                if '<' in line:
+                    matches = re.match('(.*) <(.*)>', line)
+                    if not matches:
+                        sys.exit('This line in the list of uri to parse is not formatted correctly: {line}'.format(line))
+                    feedname, line = matches.groups()
+                confobjects = line.split('|')
+                if len(confobjects) > 3 or len(confobjects) == 2:
+                    sys.exit('This line in the list of uri to parse is not formatted correctly: {line}'.format(line))
+                if len(confobjects) == 3:
+                    rss, rssobject, patternstring = line.split('|')
+                if len(confobjects) == 1:
+                    rss = confobjects[0]
+                    rssobject = ''
+                    patternstring = ''
+                # split different searched patterns
+                patterns = [i for i in patternstring.split(stringsep) if i]
+                # ignore ssl if asked
+                if ignoressl:
+                    if hasattr(ssl, '_create_unverified_context'):
+                        ssl._create_default_https_context = ssl._create_unverified_context
+                # retrieve the content of the rss
+                feed = feedparser.parse(rss)
+                if 'bozo_exception' in feed:
+                    bozoexception = True
+                    logging.warning(feed['bozo_exception'])
+                    if not accept_bozo_exceptions:
+                        continue
+                # check if the rss feed and the rss entry are valid ones
+                if 'entries' in feed:
+                    if rssobject and rssobject not in feed['entries'][0].keys():
+                        sys.exit('The rss object {rssobject} could not be found in the feed {rss}'.format(rssobject=rssobject, rss=rss))
+                else:
+                    sys.exit('The rss feed {rss} does not seem to be valid'.format(rss=rss))
+                feeds.append({'feed': feed, 'patterns': patterns, 'rssobject': rssobject, 'feedname': feedname})
+            # test if all feeds in the list were unsuccessfully retrieved and if so, leave
+            if not feeds and bozoexception:
+                sys.exit('No feed could be retrieved. Leaving.')
+    return feeds

feed2toot/feedcache.py

@@ -1,5 +1,5 @@
 # vim:ts=4:sw=4:ft=python:fileencoding=utf-8
-# Copyright © 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
@@ -38,9 +38,9 @@ class FeedCache:
             with open(self.options['cachefile']) as dbdsc:
                 dbfromfile = dbdsc.readlines()
             dblist = [i.strip() for i in dbfromfile]
-            self.dbfeed = deque(dblist, self.options['cache_limit'] )
+            self.dbfeed = deque(dblist, self.options['cache_limit'])
         else:
-            self.dbfeed = deque([], self.options['cache_limit'] )
+            self.dbfeed = deque([], self.options['cache_limit'])
 
     def append(self, rssid):
         '''Append a rss id to the cache'''

feed2toot/filterentry.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Copyright © 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
@@ -17,17 +17,11 @@
 '''Filter an entry of the RSS feeds'''
 
 # standard library imports
-from configparser import SafeConfigParser, NoOptionError, NoSectionError
-import os
-import os.path
 import sys
 
-# 3rd party library imports
-import feedparser
-
-class FilterEntry(object):
+class FilterEntry:
     '''FilterEntry class'''
-    def __init__(self, elements, entry, options, byrsspatterns, rssobject):
+    def __init__(self, elements, entry, options, byrsspatterns, rssobject, feedname):
         '''Constructor of the FilterEntry class'''
         self.matching = {}
         self.entry = entry
@@ -35,15 +29,20 @@ class FilterEntry(object):
         self.options = options
         self.byrsspatterns = byrsspatterns
         self.rssobject = rssobject
+        self.feedname = feedname
         self.main()
 
     def main(self):
         '''Main of the FilterEntry class'''
+        authorized_elements = ['feedname', ]
+        authorized_elements.extend(self.entry.keys())
         for i in self.elements:
-            if i not in self.entry:
+            if i not in authorized_elements:
                 sys.exit('The element {} is not available in the RSS feed. The available ones are: {}'.format(i, [j for j in self.entry]))
             # for the case if no pattern at all is defined
-            if not self.options['patterns'] and not self.byrsspatterns and not self.rssobject:
+            if i == 'feedname':
+                self.matching[i] = self.feedname
+            elif not self.options['patterns'] and not self.byrsspatterns and not self.rssobject:
                 self.matching[i] = self.entry[i]
             # global filter only
             elif self.options['patterns'] and not self.byrsspatterns and not self.rssobject:
@@ -67,7 +66,7 @@ class FilterEntry(object):
             if not self.options['patternscasesensitive']['{}_case_sensitive'.format(patternlist)]:
                 # not case sensitive, so we compare the lower case
                 for pattern in self.options['patterns'][patternlist]:
-                    finalpattern = pattern.lower() 
+                    finalpattern = pattern.lower()
                     finaltitle = self.entry[patternlist.split('_')[0]].lower()
                     if finalpattern in finaltitle:
                         self.matching[i] = self.entry[i]

feed2toot/hashtags.py

@@ -0,0 +1,66 @@
+# vim:ts=4:sw=4:ft=python:fileencoding=utf-8
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+'''Manage a lock file'''
+
+# standard libraires imports
+import codecs
+
+def extract_hashtags_from_list(options):
+    '''extract hashtags from the the list''' 
+    if 'hashtaglist' in options and options['hashtaglist']:
+        severalwordshashtags = codecs.open(options['hashtaglist'],
+                                           encoding='utf-8').readlines()
+        severalwordshashtags = [i.rstrip('\n') for i in severalwordshashtags]
+    else:
+        severalwordshashtags = []
+    return severalwordshashtags
+
+def build_hashtags(entry, rss, options, severalwordshashtags):
+    '''build hashtags'''
+    severalwordsinhashtag = False
+    # has the the rss feed hashtag
+    if 'tags' in entry and options['addtags']:
+        hastags = True
+    else:
+        hastags = False
+    if hastags:
+        rss['hashtags'] = []
+        for i, _ in enumerate(entry['tags']):
+            if 'hashtaglist' in options:
+                prehashtags = entry['tags'][i]['term']
+                tmphashtags = entry['tags'][i]['term']
+                for element in severalwordshashtags:
+                    if element in prehashtags:
+                        severalwordsinhashtag = True
+                        tmphashtags = prehashtags.replace(element,
+                                                          ''.join(element.split()))
+            # replace characters stopping a word from being a hashtag
+            if severalwordsinhashtag:
+                # remove ' from hashtag
+                tmphashtags = tmphashtags.replace("'", "")
+                # remove - from hashtag
+                tmphashtags = tmphashtags.replace("-", "")
+                # remove . from hashtag
+                tmphashtags = tmphashtags.replace(".", "")
+                # remove space from hashtag
+                finalhashtags = tmphashtags.replace(" ", "")
+                rss['hashtags'].append('#{}'.format(finalhashtags))
+            else:
+                nospace = ''.join(entry['tags'][i]['term'])
+                # remove space from hashtag
+                nospace = nospace.replace(" ", "")
+                rss['hashtags'].append('#{}'.format(nospace))
+    return rss

feed2toot/lock.py

@@ -0,0 +1,63 @@
+# vim:ts=4:sw=4:ft=python:fileencoding=utf-8
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+'''Manage a lock file'''
+
+# standard libraires imports
+import datetime
+import logging
+import os
+import os.path
+import sys
+
+class LockFile:
+    '''LockFile object'''
+    def __init__(self, lockfile, locktimeout):
+        '''check the lockfile and the locktimeout'''
+        self.lockfile = lockfile
+        ltimeout = datetime.timedelta(seconds=locktimeout)
+        self.lfdateformat = '%Y-%m-%d_%H-%M-%S'
+        # if a lock file exists
+        if os.path.exists(self.lockfile):
+            if os.path.isfile(self.lockfile):
+                with open(self.lockfile, 'r') as lf:
+                    lfcontent = lf.read().rstrip()
+                # lfcontent should be a datetime
+                logging.debug('Check if lock file is older than timeout ({timeout} secs)'.format(timeout=locktimeout))
+                locktime = datetime.datetime.strptime(lfcontent, self.lfdateformat)
+                if locktime < (datetime.datetime.now() - ltimeout):
+                    # remove the lock file
+                    logging.debug('Found an expired lock file')
+                    self.release()
+                    self.create_lock()
+                else:
+                    # quit because another feed2toot process is running
+                    logging.debug('Found a valid lock file. Exiting immediately.')
+                    sys.exit(0)
+        else:
+            # no lock file. Creating one
+            self.create_lock()
+
+    def create_lock(self):
+        '''Create a lock file'''
+        with open(self.lockfile, 'w') as lf:
+            currentdatestring = datetime.datetime.now().strftime(self.lfdateformat)
+            lf.write(currentdatestring)
+            logging.debug('lockfile {lockfile} created.'.format(lockfile=self.lockfile))
+
+    def release(self):
+        '''Release the lockfile'''
+        os.remove(self.lockfile)
+        logging.debug('Removed lock file.')

feed2toot/main.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # vim:ts=4:sw=4:ft=python:fileencoding=utf-8
-# Copyright © 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
@@ -21,22 +21,29 @@ import codecs
 import importlib
 import logging
 import logging.handlers
-import os
 import sys
+import re
 
-# 3rd party libraries imports
-import feedparser
+# external liraries imports
+from bs4 import BeautifulSoup
 
 # app libraries imports
 from feed2toot.addtags import AddTags
 from feed2toot.cliparse import CliParse
 from feed2toot.confparse import ConfParse
-from feed2toot.filterentry import FilterEntry
-from feed2toot.removeduplicates import RemoveDuplicates
-from feed2toot.tootpost import TootPost
 from feed2toot.feedcache import FeedCache
+from feed2toot.filterentry import FilterEntry
+from feed2toot.hashtags import build_hashtags
+from feed2toot.hashtags import extract_hashtags_from_list
+from feed2toot.lock import LockFile
+from feed2toot.message import build_message
+from feed2toot.message import send_message_dry_run
+from feed2toot.message import send_message
+from feed2toot.plugins import activate_plugins
+from feed2toot.rss import populate_rss
+from feed2toot.sortentries import sort_entries
 
-class Main(object):
+class Main:
     '''Main class of Feed2toot'''
 
     def __init__(self):
@@ -60,7 +67,7 @@ class Main(object):
         logging.debug('configured stdout level %s' % sh.level)
 
     def main(self):
-        """The main function."""
+        '''The main function'''
         clip = CliParse()
         clioptions = clip.options
         self.setup_logging(clioptions)
@@ -73,12 +80,11 @@ class Main(object):
             tweetformat = conf[2]
             feeds = conf[3]
             plugins = conf[4]
+            # check the logfile and logtimeout
+            lockfile = LockFile(options['lockfile'], options['locktimeout'])
             # create link to the persistent list
             cache = FeedCache(options)
-            if options['hashtaglist']:
-                severalwordshashtags = codecs.open(options['hashtaglist'],
-                                                   encoding='utf-8').readlines()
-                severalwordshashtags = [i.rstrip('\n') for i in severalwordshashtags]
+            severalwordshashtags = extract_hashtags_from_list(options)
             # reverse feed entries because most recent one should be sent as the last one in Mastodon
             for feed in feeds:
                 # store the patterns by rss
@@ -90,114 +96,35 @@ class Main(object):
                 if clioptions.rsssections:
                     if entries:
                         print('The following sections are available in this RSS feed: {}'.format([j for j in entries[0]]))
-                        sys.exit(0)
                     else:
-                        sys.exit('Could not parse the section of the rss feed')
-                totweet = []
-                # cache the ids of last rss feeds
-                if not clioptions.all:
-                    for i in entries:
-                        if 'id' in i and i['id'] not in cache.getdeque():
-                            totweet.append(i)
-                else:
-                    totweet = entries
-
+                        print('Could not parse the section of the rss feed')
+                        # release the lock file
+                    lockfile.release()
+                    sys.exit(0)
+                # sort entries and check if they were not previously sent
+                totweet = sort_entries(clioptions.all, cache, entries)
                 for entry in totweet:
-                    if 'id' not in entry:
-                        # malformed feed entry, skip
-                        continue
-                    logging.debug('found feed entry %s, %s', entry['id'], entry['title'])
-
-
-                    rss = {
-                        'id': entry['id'],
-                    }
-
-                    severalwordsinhashtag = False
-                    # lets see if the rss feed has hashtag
-                    if 'tags' in entry:
-                        hastags = True
-                    else:
-                        hastags = False
-
-                    if hastags:
-                        rss['hashtags'] = []
-                        for i, _ in enumerate(entry['tags']):
-                            if 'hashtaglist' in options:
-                                prehashtags = entry['tags'][i]['term']
-                                tmphashtags = entry['tags'][i]['term']
-                                for element in severalwordshashtags:
-                                    if element in prehashtags:
-                                        severalwordsinhashtag = True
-                                        tmphashtags = prehashtags.replace(element,
-                                                                          ''.join(element.split()))
-                            # replace characters stopping a word from being a hashtag
-                            if severalwordsinhashtag:
-                                # remove ' from hashtag
-                                tmphashtags = tmphashtags.replace("'", "")
-                                # remove - from hashtag
-                                tmphashtags = tmphashtags.replace("-", "")
-                                # remove . from hashtag
-                                tmphashtags = tmphashtags.replace(".", "")
-                                # remove space from hashtag
-                                finalhashtags = tmphashtags.replace(" ", "")
-                                rss['hashtags'].append('#{}'.format(finalhashtags))
-                            else:
-                                nospace = ''.join(entry['tags'][i]['term'])
-                                # remove space from hashtag
-                                nospace = nospace.replace(" ", "")
-                                rss['hashtags'].append('#{}'.format(nospace))
-
-                    elements=[]
-                    for i in tweetformat.split(' '):
-                        tmpelement = ''
-                        # if i is not an empty string
-                        if i:
-                            if i.startswith('{') and i.endswith('}'):
-                                tmpelement = i.strip('{}')
-                                elements.append(tmpelement)
-                    # match elements of the tweet format string with available element in the RSS feed
-                    fe = FilterEntry(elements, entry, options, feed['patterns'], feed['rssobject'])
+                    # populate rss with new entry to send
+                    rss = populate_rss(entry)
+                    rss = build_hashtags(entry, rss, options, severalwordshashtags)
+                    # parse tweetfomat to elements
+                    elements = re.findall(r"\{(.*?)\}",tweetformat)
+                    # strip : from elements to allow string formating, eg. {title:.20}
+                    for i,s in enumerate(elements):
+                         if s.find(':'):
+                             elements[i] = s.split(':')[0]
+                    fe = FilterEntry(elements, entry, options, feed['patterns'], feed['rssobject'], feed['feedname'])
                     entrytosend = fe.finalentry
                     if entrytosend:
-                        tweetwithnotag = tweetformat.format(**entrytosend)
-                        # remove duplicates from the final tweet
-                        dedup = RemoveDuplicates(tweetwithnotag)
-                        # only append hashtags if they exist
-                        # remove last tags if tweet too long
-                        if 'hashtags' in rss:
-                            addtag = AddTags(dedup.finaltweet, rss['hashtags'])
-                            finaltweet = addtag.finaltweet
+                        finaltweet = build_message(entrytosend, tweetformat, rss, options['tootmaxlen'], options['notagsintoot'])
+                        if clioptions.dryrun:
+                            send_message_dry_run(config, entrytosend, finaltweet)
                         else:
-                            finaltweet = dedup.finaltweet
-                        
-                    if clioptions.dryrun:
-                        if entrytosend:
-                            logging.warning('Tweet should have been sent: {tweet}'.format(tweet=finaltweet))
-                        else:
-                            logging.debug('This rss entry did not meet pattern criteria. Should have not been sent')
-                    else:
-                        storeit = True
-                        if entrytosend and not clioptions.populate:
-                            logging.debug('sending the following tweet:{tweet}'.format(tweet=finaltweet))
-                            twp = TootPost(config, finaltweet)
-                            storeit = twp.storeit()
-                        else:
-                            logging.debug('populating RSS entry {}'.format(rss['id']))
-                        # in both cas we store the id of the sent tweet
-                        if storeit:
-                            cache.append(rss['id'])
-                        # plugins
-                        if plugins and entrytosend:
-                            for plugin in plugins:
-                                capitalizedplugin = plugin.title()
-                                pluginclassname = '{plugin}Plugin'.format(plugin=capitalizedplugin)
-                                pluginmodulename = 'feed2toot.plugins.{pluginmodule}'.format(pluginmodule=pluginclassname.lower())
-                                try:
-                                    pluginmodule = importlib.import_module(pluginmodulename)
-                                    pluginclass = getattr(pluginmodule, pluginclassname) 
-                                    pluginclass(plugins[plugin], finaltweet)
-                                except ImportError as err:
-                                    print(err)
+                            send_message(config, clioptions, options, entrytosend, finaltweet, cache, rss)
+                            # plugins
+                            if plugins and entrytosend:
+                                activate_plugins(plugins, finaltweet)
             # do not forget to close cache (shelf object)
             cache.close()
+            # release the lock file
+            lockfile.release()

feed2toot/message.py

@@ -0,0 +1,82 @@
+# vim:ts=4:sw=4:ft=python:fileencoding=utf-8
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+'''Build the message'''
+
+# standard libraires imports
+import logging
+
+# external liraries imports
+from bs4 import BeautifulSoup
+
+# app libraries imports
+from feed2toot.addtags import AddTags
+from feed2toot.removeduplicates import RemoveDuplicates
+from feed2toot.tootpost import TootPost
+
+def build_message(entrytosend, tweetformat, rss, tootmaxlen, notagsintoot):
+    '''populate the rss dict with the new entry'''
+    tweetwithnotag = tweetformat.format(**entrytosend)
+    # replace line breaks
+    tootwithlinebreaks = tweetwithnotag.replace('\\n', '\n')
+    # remove duplicates from the final tweet
+    dedup = RemoveDuplicates(tootwithlinebreaks)
+    # only add tags if user wants to
+    if not notagsintoot:
+        # only append hashtags if they exist
+        # remove last tags if tweet too long
+        if 'hashtags' in rss:
+            addtag = AddTags(dedup.finaltweet, rss['hashtags'])
+            finaltweet = addtag.finaltweet
+        else:
+            finaltweet = dedup.finaltweet
+    else:
+        finaltweet = dedup.finaltweet
+    # strip html tags
+    finaltweet = BeautifulSoup(finaltweet, 'html.parser').get_text()
+    # truncate toot to user-defined value whatever the content is
+    if len(finaltweet) > tootmaxlen:
+        finaltweet = finaltweet[0:tootmaxlen-1]
+        return ''.join([finaltweet[0:-3], '...'])
+    else:
+        return finaltweet
+
+def send_message_dry_run(config, entrytosend, finaltweet):
+    '''simulate sending message using dry run mode'''
+    if entrytosend:
+        logging.warning('Would toot with visibility "{visibility}": {toot}'.format(
+            toot=finaltweet,
+            visibility=config.get(
+                'mastodon', 'toot_visibility',
+                fallback='public')))
+    else:
+        logging.debug('This rss entry did not meet pattern criteria. Should have not been sent')
+
+def send_message(config, clioptions, options, entrytosend, finaltweet, cache, rss):
+    '''send message'''
+    storeit = True
+    if entrytosend and not clioptions.populate:
+        logging.debug('Tooting with visibility "{visibility}": {toot}'.format(
+            toot=finaltweet,
+            visibility=config.get(
+                'mastodon', 'toot_visibility',
+                fallback='public')))
+        twp = TootPost(config, options, finaltweet)
+        storeit = twp.storeit()
+    else:
+        logging.debug('populating RSS entry {}'.format(rss['id']))
+    # in both cas we store the id of the sent tweet
+    if storeit:
+        cache.append(rss['id'])

feed2toot/plugins/__init__.py

@@ -13,3 +13,19 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+# standard libraires imports
+import importlib
+
+def activate_plugins(plugins, finaltweet):
+    '''activate plugins'''
+    for plugin in plugins:
+        capitalizedplugin = plugin.title()
+        pluginclassname = '{plugin}Plugin'.format(plugin=capitalizedplugin)
+        pluginmodulename = 'feed2toot.plugins.{pluginmodule}'.format(pluginmodule=pluginclassname.lower())
+        try:
+            pluginmodule = importlib.import_module(pluginmodulename)
+            pluginclass = getattr(pluginmodule, pluginclassname)
+            pluginclass(plugins[plugin], finaltweet)
+        except ImportError as err:
+            print(err)

feed2toot/plugins/influxdbplugin.py

@@ -16,9 +16,6 @@
 # Push values to a influxdb database
 '''Push values to a influxdb database'''
 
-# standard libraries imports
-import json
-
 # 3rd party libraries imports
 from influxdb import InfluxDBClient
 
@@ -30,13 +27,16 @@ class InfluxdbPlugin(object):
         self.data = data
         self.datatoinfluxdb = []
         self.client = InfluxDBClient(self.plugininfo['host'],
-                                self.plugininfo['port'],
-                                self.plugininfo['user'],
-                                self.plugininfo['pass'],
-                                self.plugininfo['database'])
+            self.plugininfo['port'],
+            self.plugininfo['user'],
+            self.plugininfo['pass'],
+            self.plugininfo['database'])
         self.main()
 
     def main(self):
         '''Main of the PiwikModule class'''
-        self.datatoinfluxdb.append({'measurement': self.plugininfo['measurement'], 'fields': {'value': self.data}})
+        self.datatoinfluxdb.append({
+            'measurement': self.plugininfo['measurement'],
+            'fields': {'value': self.data}
+        })
         self.client.write_points(self.datatoinfluxdb)

feed2toot/removeduplicates.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # vim:ts=4:sw=4:ft=python:fileencoding=utf-8
-# Copyright © 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
@@ -17,7 +17,7 @@
 # Remove duplicates from the final string before sending the tweet
 '''Remove duplicates from the final string before sending the tweet'''
 
-class RemoveDuplicates(object):
+class RemoveDuplicates:
     '''Remove duplicates from the final string before sending the tweet'''
     def __init__(self, tweet):
         '''Constructor of RemoveDuplicates class'''
@@ -32,12 +32,12 @@ class RemoveDuplicates(object):
             if element != ' ' and (element.startswith('http://') or element.startswith('https://')):
                 newlink = True
                 # if we already found this link, increment the counter
-                for i,_ in enumerate(links):
+                for i, _ in enumerate(links):
                     if links[i]['link'] == element:
                         newlink = False
                         links[i]['count'] += 1
                 if newlink:
-                    links.append({'link': element, 'count': 1}) 
+                    links.append({'link': element, 'count': 1})
         # remove duplicates
         validatedlinks = []
         for i in range(len(links)):
@@ -45,14 +45,14 @@ class RemoveDuplicates(object):
                 validatedlinks.append(links[i])
         wildcard = 'FEED2TOOTWILDCARD'
         for element in validatedlinks:
-            for i in range(element['count']): 
+            for i in range(element['count']):
                 # needed for not inversing the order of links if it is a duplicate
                 # and the second link is not one
                 if i == 0:
-                    self.tweet = self.tweet.replace(element['link'], wildcard, 1 )
+                    self.tweet = self.tweet.replace(element['link'], wildcard, 1)
                 else:
                     self.tweet = self.tweet.replace(element['link'], '', 1)
-            # finally 
+            # finally
             self.tweet = self.tweet.replace(wildcard, element['link'], 1)
         # remove all 2xspaces
         self.tweet = self.tweet.replace('  ', ' ')

feed2toot/rss.py

@@ -0,0 +1,42 @@
+# vim:ts=4:sw=4:ft=python:fileencoding=utf-8
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+'''Manage a lock file'''
+
+# standard libraires imports
+import datetime
+import logging
+import os
+import os.path
+import sys
+
+def populate_rss(entry):
+    '''populate the rss dict with the new entry'''
+    if 'id' in entry:
+        logging.debug('found feed entry {entryid}'.format(entryid=entry['id']))
+        rss = {
+            'id': entry['id'],
+        }
+    elif 'guid' in entry:
+        logging.debug('found feed entry {entryid}'.format(entryid=entry['guid']))
+        rss = {
+            'id': entry['guid'],
+        }
+    else:
+        logging.debug('found feed entry {entryid}'.format(entryid=entry['link']))
+        rss = {
+            'id': entry['link'],
+        }
+    return rss

feed2toot/sortentries.py

@@ -0,0 +1,42 @@
+# vim:ts=4:sw=4:ft=python:fileencoding=utf-8
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+
+'''Manage a lock file'''
+
+# standard libraires imports
+import datetime
+import logging
+import os
+import os.path
+import sys
+
+def sort_entries(is_all, cache, entries):
+    '''sort entries before sending'''
+    totweet = []
+    if not is_all:
+        for i in entries:
+            if 'id' in i:
+                if i['id'] not in cache.getdeque():
+                    totweet.append(i)
+            elif 'guid' in i:
+                if i['guid'] not in cache.getdeque():
+                    totweet.append(i)
+            else:
+                # if id or guid not in the entry, use link
+                if i['link'] not in cache.getdeque():
+                    totweet.append(i)
+    else:
+        totweet = entries
+    return totweet

feed2toot/tootpost.py

@@ -1,5 +1,5 @@
 # vim:ts=4:sw=4:ft=python:fileencoding=utf-8
-# Copyright © 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
@@ -15,24 +15,16 @@
 
 """Checks an RSS feed and posts new entries to Mastodon."""
 
-# standard libraires imports
-from configparser import SafeConfigParser, NoOptionError, NoSectionError
-from argparse import ArgumentParser
-import codecs
-import logging
-import os
-import sys
-
 # 3rd party libraries imports
-import feedparser
 from mastodon import Mastodon
 
 class TootPost:
     '''TootPost class'''
 
-    def __init__(self, config, toot):
+    def __init__(self, config, options, toot):
         '''Constructore of the TootPost class'''
         self.config = config
+        self.options = options
         self.store = True
         self.toot = toot
         self.main()
@@ -40,11 +32,16 @@ class TootPost:
     def main(self):
         '''Main of the TweetPost class'''
         mastodon = Mastodon(
-            client_id = self.config.get('mastodon', 'client_credentials'),
-            access_token = self.config.get('mastodon', 'user_credentials'),
-            api_base_url = self.config.get('mastodon', 'instance_url')
+            client_id=self.config.get('mastodon', 'client_credentials'),
+            access_token=self.config.get('mastodon', 'user_credentials'),
+            api_base_url=self.config.get('mastodon', 'instance_url')
         )
-        mastodon.toot(self.toot)
+        toot_visibility = self.config.get('mastodon', 'toot_visibility', fallback='public')
+        if 'custom' in self.options['media']:
+            mediaid = mastodon.media_post(self.config['media']['custom'])
+            mastodon.status_post(self.toot, media_ids=[mediaid], visibility=toot_visibility)
+        else:
+            mastodon.status_post(self.toot, visibility=toot_visibility)
 
     def storeit(self):
         '''Indicate if the tweet should be stored or not'''

scripts/__init__.py

@@ -0,0 +1,15 @@
+#!/usr/bin/env python3
+# vim:ts=4:sw=4:ft=python:fileencoding=utf-8
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>

scripts/feed2toot

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 # vim:ts=4:sw=4:ft=python:fileencoding=utf-8
-# Copyright © 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or

scripts/register_feed2toot_app

@@ -1,7 +1,6 @@
 #!/usr/bin/env python3
-#!/usr/bin/env python3
 # vim:ts=4:sw=4:ft=python:fileencoding=utf-8
-# Copyright © 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright © 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
@@ -15,16 +14,42 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>
 
+from argparse import ArgumentParser
 from getpass import getpass
 from os import getcwd
+from os import linesep
+from os import sep
 from mastodon import Mastodon
 from mastodon.Mastodon import MastodonIllegalArgumentError
 import sys
 
-print('\nThis app generates Mastodon app credentials needed by Feed2toot.\nfeed2toot_clientcred.txt and feed2toot_usercred.txt will be written in the current dir {cwd}.\nOne connection is initiated to create the app.\nYour password is *not* stored.\n'.format(cwd=getcwd()))
+__version__ = '0.2'
+
+epilog = 'For more information: https://feed2toot.readthedocs.io'
+description = 'Create a Mastodon app for Feed2toot'
+parser = ArgumentParser(prog='register_feed2toot_app',
+                        description=description,
+                        epilog=epilog)
+parser.add_argument('--version', action='version', version=__version__)
+parser.add_argument('--client-credentials-file', dest='clientcredfile', help='the name of the client credentials for the Mastodon app', default='feed2toot_clientcred.txt')
+parser.add_argument('--user-credentials-file', dest='usercredfile', help='the name of the user credentials for the Mastodon app', default='feed2toot_usercred.txt')
+parser.add_argument('--name', help='the name of the Mastodon app', default='feed2toot')
+parser.add_argument('--instance', help='the URL of the Mastodon instance')
+parser.add_argument('--username', help='the username of your Mastodon account')
+parser.add_argument('--password', help='the password of your Mastodon account')
+opts = parser.parse_args()
+
+clientcredfile=opts.clientcredfile
+usercredfile=opts.usercredfile
+
+headline = '{linesep}This script generates the Mastodon application credentials for Feed2toot.{linesep}{clientcredfile} and {usercredfile} will be written{linesep}in the current directory: {cwd}.{linesep}WARNING: previous files with the same names will be overwritten.{linesep}{linesep}A connection is also initiated to create the application.{linesep}Your password is *not* stored.{linesep}'.format(linesep=linesep, clientcredfile=clientcredfile, usercredfile=usercredfile, cwd=getcwd())
+print(headline)
+
 
 # get the instance
-instance = input('Mastodon instance url (defaults to https://mastodon.social):')
+instance = opts.instance
+if not instance:
+    instance = input('Mastodon instance URL (defaults to https://mastodon.social): ')
 if not instance:
     instance = 'https://mastodon.social'
 elif not instance.startswith('http'):
@@ -32,34 +57,45 @@ elif not instance.startswith('http'):
 
 # get the username
 userok = False
+quit_on_error = True
 while not userok:
-    user = input('Mastodon login:')
+    user = opts.username
     if not user:
-        print('Your Mastodon username can not be empty')
-        userok = False
+        user = input('Mastodon login: ')
+        quit_on_error = False
+    if not user:
+        print('Your Mastodon username can not be empty.')
     elif '@' not in user or '.' not in user:
-        print('Your Mastodon username should be an email')
-        userok = False
+        print('Your Mastodon username should be an email.')
     else:
         userok = True
+    if not userok and quit_on_error:
+        exit()
 
 # get the password
-password = getpass(prompt='Mastodon password:')
+password = opts.password
+if not password:
+    password = getpass(prompt='Mastodon password: ')
 Mastodon.create_app(
-     'feed2toot',
+      opts.name,
       api_base_url=instance,
-      to_file = '{cwd}/feed2toot_clientcred.txt'.format(cwd=getcwd())
+      to_file = '{cwd}{sep}{clientcredfile}'.format(cwd=getcwd(), sep=sep, clientcredfile=clientcredfile)
 )
-mastodon = Mastodon(client_id = '{cwd}/feed2toot_clientcred.txt'.format(cwd=getcwd()),
+mastodon = Mastodon(client_id = '{cwd}{sep}{clientcredfile}'.format(cwd=getcwd(), sep=sep, clientcredfile=clientcredfile),
                     api_base_url=instance)
 try:
     mastodon.log_in(
         user,
         password,
-        to_file = '{cwd}/feed2toot_usercred.txt'.format(cwd=getcwd())
+        to_file = '{cwd}{sep}{usercredfile}'.format(cwd=getcwd(), sep=sep, usercredfile=usercredfile)
     )
 except MastodonIllegalArgumentError as err:
     print(err)
-    sys.exit('\nMy guess is bad login/password\n')
-print('The feed2toot app was added to your preferences=>authorized apps page')
+    sys.exit('{linesep}I guess you entered a bad login or password.{linesep}'.format(linesep=linesep))
+
+summary = '{linesep}The app {appname} was added to your Preferences=>Accounts=>Authorized apps page.{linesep}The file {clientcredfile} and {usercredfile} were created in the current directory.{linesep}'.format(appname=opts.name,
+              linesep=linesep,
+              clientcredfile=clientcredfile,
+              usercredfile=usercredfile)
+print(summary)
 sys.exit(0)

setup.py

@@ -1,4 +1,4 @@
-# Copyright 2015-2017 Carl Chenet <carl.chenet@ohmytux.com>
+# Copyright 2015-2021 Carl Chenet <carl.chenet@ohmytux.com>
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
@@ -25,24 +25,24 @@ CLASSIFIERS = [
     'Operating System :: POSIX :: Linux',
     'Programming Language :: Python :: 3.4',
     'Programming Language :: Python :: 3.5',
-    'Programming Language :: Python :: 3.6'
+    'Programming Language :: Python :: 3.6',
+    'Programming Language :: Python :: 3.7'
 ]
 
-
 setup(
     name='feed2toot',
-    version='0.3',
+    version='0.17',
     license='GNU GPL v3',
-    description='Parse rss feed and tweet new posts to Mastodon',
-    long_description='Parse rss feed and tweet new posts to the Mastodon social network',
+    description='Parse rss feeds and send new posts to Mastodon',
+    long_description='Parse rss feeds and send new posts to the Mastodon social network',
     author = 'Carl Chenet',
-    author_email = 'chaica@ohmytux.com',
-    url = 'https://github.com/chaica/feed2toot',
+    author_email = 'carl.chenet@ohmytux.com',
+    url = 'https://gitlab.com/chaica/feed2toot',
     classifiers=CLASSIFIERS,
-    download_url='https://github.com/chaica/feed2toot',
+    download_url='https://gitlab.com/chaica/feed2toot',
     packages=find_packages(),
     scripts=['scripts/feed2toot', 'scripts/register_feed2toot_app'],
-    install_requires=['feedparser', 'Mastodon.py'],
+    install_requires=['beautifulsoup4', 'feedparser', 'Mastodon.py'],
     extras_require={
         'influxdb':  ["influxdb"]
     }