Tools that have been useful to me in the past, or might be useful in the future, loosely grouped. Not things I use daily.
🦋 Web scrapers
Curl impersonate
A special build of curl that can impersonate the four major browsers: Chrome, Edge, Safari & Firefox. curl-impersonate is able to perform TLS and HTTP handshakes that are identical to that of a real browser.
When you use an HTTP client with a TLS website, it first performs a TLS handshake. The first message of that handshake is called Client Hello. The Client Hello message that most HTTP clients and libraries produce differs drastically from that of a real browser.
If the server uses HTTP/2, then in addition to the TLS handshake there is also an HTTP/2 handshake where various settings are exchanged. The settings that most HTTP clients and libraries use differ as well from those of any real browsers.
For these reasons, some web services use the TLS and HTTP handshakes to fingerprint which client is accessing them, and then present different content for different clients. These methods are known as TLS fingerprinting and HTTP/2 fingerprinting respectively. Their widespread use has led to the web becoming less open, less private and much more restrictive towards specific web clients
With the modified curl in this repository, the TLS and HTTP handshakes look exactly like those of a real browser.
Puppeteer heap snapshot
puppeteer-heap-snapshot is a Node.js module that, given a Puppeteer browser page, can capture and parse a heap snapshot and deserialize objects that contain a set of properties. It comes with a nifty CLI tool too so we can quickly prototype scrapers from our terminal.
Instead of trying to use CSS selectors to get at the data we want, we grab the data straight out of the browser’s working memory.
grab-site is an easy preconfigured web crawler designed for backing up websites. Give grab-site a URL and it will recursively crawl the site and write WARC files. Internally, grab-site uses a fork of wpull for crawling.
Note
Still looking for something that can maintain an archive of a Facebook group.
🐌 Convert raster to vector (unsorted)
https://www.vectorization.org/
https://www.visioncortex.org/vtracer/
https://online.rapidresizer.com/tracer.php
https://fconvert.com/autotrace/
https://online-converting.com/autotrace/
https://github.com/fromtheexchange/image2svg-awesome
📈 Convert text to diagram
JavaScript based diagramming and charting tool that renders Markdown-inspired text definitions to create and modify diagrams dynamically.
Not the prettiest output, but the dot language makes generating large diagrams really simple. I use it whenever I need to parse structure (eg library dependencies) into a graphical overview.
Consumes dot files and lets you query them. What’s the shortest path between A and B? What relies on C?
A more complete list.
🔪 Edit PDFs
This is absolutely fantastic. A whole bunch of PDF tools, slapped in a Docker container with a simple and effective web interface on top. Less than five minutes to get running. I used it to generate my John Pory PDF.
🦀 Shell scripting
A collection of useful utilities for enhancing shell scripts.
Dotfile (Unix config file) manager. Maybe it’s better than a github repo, maybe it’s just another yak waiting to be shaved.
Set environment variables based on path.
🦚 Graphics libraries
Rough.js is a small (<9kB gzipped) graphics library that lets you draw in a sketchy, hand-drawn-like, style. The library defines primitives to draw lines, curves, arcs, polygons, circles, and ellipses. It also supports drawing SVG paths.
Programming
Code editor. Clean. Fast. Configurable key bindings. I don’t use one tenth of the features of an IDE, so this might be perfect for me.
🌀 Other
Battery charge limiter for macs.
Makes your MBP sound like a clicky-clacky buckling spring keyboard. I know it sounds like a joke. It probably is a joke. But for some reason it really helps me focus.
Enhance the capabilities of Amazon’s Fire tablets.