DFSG NEW review for python-newspaper version 0.2.8-1

Summary

• Licensing is DFSG-compliant and debian/copyright is accurate: No
• Packaging is of good quality: No
• Issues: 0 blocker, 1 important, 3 minor
• debian/watch: version 5
• debian/copyright: machine-readable
• Scan before the review: nothing found

Licensing

All files carry free licences: MIT/Expat for the newspaper code, Apache-2.0 for the two python-goose-derived modules and for the stopword lists. Two points make debian/copyright not fully accurate: the specific stanza for newspaper/extractors.py/parsers.py overrides the catch-all and drops Lucas Ou-Yang, whose copyright notice is in extractors.py itself; and the Apache copyright over all stopword lists rests on a guess, since only stopwords-fr.txt carries an ASF header. No DFSG violation was found.

Packaging

The packaging is close: DEP-5 copyright, compat 14, dh with pybuild, py3dist-overrides for jieba, salsa-ci and gbp configuration. Two things need fixing: the upstream requirements that feed install_requires still name tinysegmenter, which the Debian patch removes and which has no Debian package, plus unused feedfinder2/PyYAML; and the Vcs-Browser URL is a typo pointing at a non-existent repository.

Issues

1. [important] [packaging] install_requires still lists tinysegmenter (removed by a patch) and other unused packages

• Files: requirements.txt:13, requirements.txt:3, requirements.txt:10, newspaper3k.egg-info/requires.txt:13, debian/patches/replace-tinysegmenter-by-icu.patch
• Basis: checked, in the package or with a reference tool
• Evidence: requirements.txt: tinysegmenter==0.3, PyYAML>=3.11, feedfinder2>=0.0.4; setup.py does install_requires=required from that file; the patch removes the only import tinysegmenter.
• Problem: debian/patches/replace-tinysegmenter-by-icu.patch replaces the tinysegmenter use with PyICU, yet the package metadata still requires it. dh_python3(1) translates requires.txt/Requires-Dist into ${python3:Depends}; apt-cache show python3-tinysegmenter finds no such binary, so the generated dependency is at best an unmapped requirement with a build warning and a phantom dependency, at worst unsatisfiable. PyYAML and feedfinder2 are not imported anywhere in newspaper/ either, so the binary declares dependencies on code it never uses.
• Fix: Add a patch dropping tinysegmenter, feedfinder2 and PyYAML from requirements.txt (or replace the install_requires with an explicit list), so that the installed metadata and the generated Depends match the code.

2. [minor] [licensing] Copyright stanza for extractors.py/parsers.py drops Lucas Ou-Yang's notice

• Files: debian/copyright:9-12, newspaper/extractors.py:8-12
• Basis: checked, in the package or with a reference tool
• Evidence: debian/copyright: Files: newspaper/parsers.py newspaper/extractors.py / Copyright: Xavier Grangier; newspaper/extractors.py: __author__ = 'Lucas Ou-Yang', __copyright__ = 'Copyright 2014, Lucas Ou-Yang'.
• Problem: Later stanzas override Files: *, so for these two files the only declared holder is Xavier Grangier. The files themselves carry Lucas Ou-Yang's copyright notice; a machine-readable copyright that omits a holder present in the file is inaccurate (copyright-format 1.0 §5.2, Copyright field).
• Fix: Add the file's own holder, e.g. Copyright: 2013 Xavier Grangier, 2014 Lucas Ou-Yang.

3. [minor] [licensing] ASF copyright asserted for every stopword list on an admitted guess

• Files: debian/copyright:14-18, newspaper/resources/text/stopwords-fr.txt:1-14
• Basis: checked, in the package or with a reference tool
• Evidence: Copyright: 2001-2025 The Apache Software Foundation with Comment: Most stopwords are probably taken from Lucene's StopAnalyzer; only stopwords-fr.txt carries the ASF header among the 36 stopwords-*.txt files.
• Problem: The Copyright field must state the actual holders. The comment concedes that the provenance of most lists was not established, and 35 of the files bear no licence or copyright notice at all, so the ASF attribution is unverified and may be wrong for lists taken from elsewhere.
• Fix: Establish where each stopword list came from (e.g. Lucene, NLTK, or another project) and either adjust the stanza or record the provenance per group.

4. [minor] [packaging] Vcs-Browser URL is a typo

• Files: debian/control:12
• Basis: checked, in the package or with a reference tool
• Evidence: Vcs-Browser: https://salsa.debian.org/python-team/packages/python3-newpaper while Vcs-Git: .../python3-newspaper.git
• Problem: newpaper is missing an s; the URL points at a non-existent repository.
• Fix: Use https://salsa.debian.org/python-team/packages/python3-newspaper.

Verified

• Read debian/copyright in full: DEP-5 1.0, correct Format: URL, stanzas for *, the two goose modules, resources/text/* and debian/*; the Expat text is complete and Apache-2.0 points to /usr/share/common-licenses/Apache-2.0.
• The pre-scan's author check found no missing holder among the 74 scanned files; the only holders in the tree are Lucas Ou-Yang and Xavier Grangier, both named (see the stanza issue above).
• Every module in newspaper/ carries __license__ = 'MIT' and __copyright__ = 'Copyright 2014, Lucas Ou-Yang'; newspaper/videos/* and newspaper/cleaners.py carry no separate notice and are covered by Files: *.
• newspaper/resources/text/stopwords-fr.txt is the only stopword list with an Apache header; the other lists and the resources/misc/ files (google_sources.txt, popular_sources.txt, useragents.txt, stopwords-nlp-en.txt) are plain data under the catch-all stanza.
• No binaries, minified JS/CSS, PDFs, archives or bytecode in the tree; newspaper3k.egg-info/ and PKG-INFO are upstream-generated sdist metadata whose source of record (setup.py, setup.cfg) is present.
• Packaging files read: control, rules, changelog, source/format (3.0 quilt), both patches with DEP-3 headers, patches/series, gbp.conf, upstream/metadata, salsa-ci.yml, py3dist-overrides; no maintainer scripts or autopkgtest suite exist.
• Standards-Version: 4.7.4 is current enough against Policy 4.7.4.1 (policy tool); compat 14 is debhelper's recommended level, and dh-sequence-single-binary is a valid explicit activation at compat 14 (debhelper tool).
• Build-Depends names packages that exist: python3-jieba, python3-lxml-html-clean and python3-icu checked with apt-cache; ${python3:Depends} is not needed in control since compat 14 makes dh_gencontrol add relationship substvars itself.
• Runtime imports match the declared build/runtime dependencies: requests, feedparser, PIL, bs4, cssselect, dateutil, tldextract, lxml, nltk, jieba and icu are all imported by newspaper/; python3-six is build-depended but not imported, which is harmless.
• uscan found the packaged version as the newest upstream release (upstream 0.2.8, packaged 0.2.8).
• Every pointer of debian/copyright to /usr/share/common-licenses names a file that is there (Apache-2.0).

Not confirmed

• newspaper/cleaners.py (DocumentCleaner) and newspaper/videos/extractors.py (VideoExtractor, provider list youtube, vimeo, dailymotion, kewego) look like python-goose code as well; if so, the Apache-2.0 stanza should cover them too. The README only says newspaper uses "a lot of python-goose's parsing code" and the referenced GOOSE-LICENSE.txt is absent from the tarball, so the full extent of the goose-derived files cannot be checked from the package.
• debian/rules sets PYBUILD_SYSTEM=pyproject and Build-Depends on pybuild-plugin-pyproject, but the source has no pyproject.toml (only setup.py/setup.cfg). A build would show whether pybuild's pyproject plugin falls back to the setuptools legacy path; pybuild is not installed on the reviewing system, so this could not be checked.
• What dh_python3 does with a requirement it cannot map to a Debian package (tinysegmenter) could not be checked because dh-python is not installed on the reviewing system; the source-side defect (stale requirement) is reported regardless.

Proposed changes

None.