0%

常用开源项目盘点

介绍一些常用的优秀开源项目,比如用来下载网络视频的 you-get、爬取 Pixiv 图片的 PixivUtil2、能够获取直链的自建网盘 FileBrowser。


Python的安装与配置

因为常用的开源项目中 Python 项目居多,所以简单介绍下 Python 项目在 Windows 系统下的使用方法。

下载安装

下载前先检测当前系统中是否已经安装了 Python 和 Pip。命令行输入:

1
2
3
4
5
$ python -V
Python 3.6.9

$ pip -V
pip 21.0.1 from /home/sannaha/.local/lib/python3.6/site-packages/pip (python 3.6)

如果没有显示出版本信息,则需要下载。

Windows

打开 官网下载 Python 安装包,Python 2.7.9+Python 3.4+ 以上版本都自带 pip 工具,安装时注意勾选 Add Python 3.x to PATH

Linux

以 ubuntu 18.04 为例进行演示。

1
2
3
4
5
6
7
8
9
10
# 卸载python2
$ sudu apt-get remove python
# 安装python3
$ sudo apt-get install python3

# 强制建立软链接,将python指向python3
$ sudo ln -sf /usr/bin/python3.6 /usr/bin/python

# 安装pip
$ sudo apt-get install python-pip

配置镜像源

pip

如果是 Windows 环境,在 %USERPROFILE% 下创建 pip 文件夹,并在里面新建配置文件 pip.ini,文件内容如下:

1
2
3
4
[global]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple
[install]
trusted-host=mirrors.aliyun.com

如果是 Linux 环境,创建 ~/.pip 目录,并在里面新建配置文件 pip.conf,文件内容同上。

apt

修改 apt 源为阿里源。源列表文件的内容需要与系统版本一致,这里以 ubuntu 18.04 为例进行演示,其版本代号为 bionic。查看当前系统的版本代号:

1
2
$ lsb_release -c
Codename: bionic
  1. 备份源列表文件 sources.list

    1
    $ sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
  2. 编辑源列表文件:

    1
    $ sudo vi /etc/apt/sources.list
  3. 替换内容:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    # ubuntu 18.04(bionic)
    deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
    deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse

    deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
    deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse

    deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
    deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse

    deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
    deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse

    deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
    deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
  4. 更新软件列表:

    1
    $ sudo apt-get update

其他版本:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# ubuntu 14.04(trusty)
deb https://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse

deb https://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse

deb https://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse

# ubuntu 16.04(xenial)
deb http://mirrors.aliyun.com/ubuntu/ xenial main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial main

deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates main

deb http://mirrors.aliyun.com/ubuntu/ xenial universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates universe

deb http://mirrors.aliyun.com/ubuntu/ xenial-security main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security main
deb http://mirrors.aliyun.com/ubuntu/ xenial-security universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security universe

# ubuntu 20.04(focal)
deb http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse

参考资料
Ubuntu 镜像 - 阿里云

安装依赖

在一个 Python 项目通常会依赖其他项目,会以 requirements.txt 文件的形式描述这些依赖:

1
$ pip install -r requirements.txt

you-get下载网络视频

you-get 是一个命令行程序,用于下载网络上的媒体信息,如 Bilibili 视频、YouTube 视频:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 安装you-get
$ pip3 install you-get

# 升级you-get
$ pip3 install --upgrade you-get

# 下载B站视频(带xml弹幕文件)
$ you-get https://www.bilibili.com/video/BV1rp4y1e745

# 查看目标视频的可用画质与格式
$ you-get -i https://www.bilibili.com/video/BV1rp4y1e745

# 下载指定画质与格式的视频
$ you-get --format=dash-hdflv2_4k https://www.bilibili.com/video/BV1rp4y1e745

# 下载YouTube视频(使用代理,带srt字幕文件)
$ you-get -x 127.0.0.1:1080 'https://www.youtube.com/watch?v=XKZM_AjCUr0'

you-get-youtube

参考资料
中文说明


PixivUtil2爬虫

根据作品 ID、用户 ID、个人收藏、TAG 以及收藏数爬取 Pixiv 的插画。

安装

1
2
3
4
5
6
# 下载PixivUtil2
$ git clone https://github.com/Nandaka/PixivUtil2.git
# 安装依赖
$ pip install -r requirements.txt
# 运行
$ py PixivUtil2.py

配置

PixivUtil2 通过文件 config.ini 配置相关功能。

配置登录验证:

1
2
3
4
5
6
[Authentication]
username = <username>
password = <password>
cookie = <PHPSHESSID>
cookiefanbox =
refresh_token =

配置网络代理:

1
2
3
4
5
6
7
8
9
10
11
12
[Network]
useproxy = True
proxyaddress = http://127.0.0.1:1080
useragent = Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36
userobots = True
timeout = 60
retry = 3
retrywait = 5
downloaddelay = 2
checknewversion = True
opennewversion = True
enablesslverification = True

配置文件命名规则:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[Filename]
filenameformat = %artist%(%member_id%)\%artist%(%member_id%)-%urlFilename%-%title%
filenamemangaformat = %artist%(%member_id%)\%artist%(%member_id%)-%urlFilename%-%title%
filenameinfoformat = %artist%(%member_id%)\%artist%(%member_id%)-%urlFilename%-%title%
filenamemangainfoformat = %artist%(%member_id%)\%artist%(%member_id%)-%urlFilename%-%title%
filenameseriesjson = %artist%(%member_id%)\%artist%(%member_id%)-%manga_series_id%-%manga_series_title%
filenameformatsketch = %artist%(%member_id%)\%artist%(%member_id%)-%urlFilename%-%title%
filenameformatnovel = %artist%(%member_id%)-\%artist%(%member_id%)%manga_series_id%-%manga_series_order%-%urlFilename%-%title%
avatarnameformat =
backgroundnameformat =
tagsseparator = ,
createmangadir = False
usetagsasdir = False
urldumpfilename = url_list_%Y%m%d
usetranslatedtag = False
tagtranslationlocale = en
custombadchars =

配置存储路径:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[Settings]
downloadlistdirectory = .
uselist = True
processfromdb = True
rootdirectory = ../Pixiv收藏/
downloadavatar = False
usesuppresstags = False
tagslimit = -1
writeimageinfo = False
writeimagejson = False
writerawjson = False
rawjsonfilter = id,title,description,alt,userIllusts,storableTags,zoneConfig,extraData,comicPromotion,fanboxPromotion
includeseriesjson = False
verifyimage = False
writeurlindescription = False
urlblacklistregex =
dbpath =
setlastmodified = True
uselocaltimezone = False

自建网盘获取直链

File Browser 是一个 GO 语言编写的轻量级 WEB 文件管理器,提供了文件上传、下载、删除、预览、重命名、分享等诸多功能,并支持多用户管理。

![File Browser-功能展示](/ExcellentProject/File Browser-功能展示.gif)

搭建

解压从 release 页面 下载的打包好的压缩包即可得到我们需要的可执行文件 filebrowser

部署命令:

1
2
# 指定服务器的/webdisk作为网盘根目录,filebrowser.db存放用户信息,使用9999端口
./filebrowser -a 0.0.0.0 -p 9999 -r /netdisk -d /netdisk/filebrowser.db

可以使用 systemd 新建一个开机启动的守护进程服务,保持后台常驻。在 /etc/systemd/system 下新建 fb.service 文件:

/etc/systemd/system/fb.service
1
2
3
4
5
6
7
8
9
10
11
12
[Unit]
Description=filebrowser service
After=network.target

[Service]
Type=simple
Restart=always

ExecStart=/server/filebrowser/filebrowser -a 0.0.0.0 -p 9999 -r /netdisk -d /netdisk/filebrowser.db

[Install]
WantedBy=multi-user.target

使服务生效并启动:

1
2
systemctl enable fb.service
systemctl start fb.service

Nginx 413错误

上传文件时,Nginx 报错提示 413 Request Entity Too Large。原因是 Nginx 判断客户端发送的 HTTP 数据流过大,可以在 nginx.conf 中配置 client_max_body_size 值,默认为 1M,显然无法满足作为网盘的需求,可以根据上传文件的大小进行调整:

nginx.conf
1
2
3
4
5
6
7
8
9
10
11
12
http {
...

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
+ client_max_body_size 100m;

server {...}
}

参考资料
** VPS 上搭建简易的私有云盘
**