You know that feeling when you’re trying to find a post (skeet) that you’ve seen some time ago, and you can’t find it? It happens to me all the time. Let’s say someone from the Bluesky dev team wrote some piece of technical detail, or some ATProto developer from the community posted about their project - I know more or less what I’m looking for, but not enough to find it in the global search. But I know I probably gave it a like, because I like everything 😅
It would be nice if Bluesky allowed you to search for text in your likes, like e.g. Mastodon does (the in:library modifier), but it doesn’t support that yet. But “we can just do things”, right?
I’ve seen a project before that attempted to do this client-side, fetching all your likes and searching in them in memory, but this isn’t quite realistic for someone like me who has around… 130 thousand likes from 2.5 years 🫣 So I figured this needs to be done on a server.
So I’ve built a service I named Lycan 🐺 - it’s kind of like a tiny specialized AppView, which only indexes some specific things from some specific people. To avoid having to keep a basically full-network AppView with a multi-TB database, it only indexes posts and likes on demand from people who request to use it. So the first time you want to use it, you need to ask it to run an import process, which happens on the server and can take anything between a few minutes and an hour, depending on how much data there is to download. After that, new likes are being indexed live from the firehose (and they’re deleted if you either unlike a post, or if the post itself is deleted).
At the moment, Lycan indexes four types of content:
- posts you’ve liked
- posts you’ve reposted
- posts you’ve quoted
- your old-style bookmarks (using the 📌 emoji method)
New bookmarks are private data, and I will need to use a different approach for that (OAuth).
Lycan is written in Ruby (like almost all of my ATProto code 😎), using Sinatra and ActiveRecord, with Postgres as the database. I’m running it at lycan.feeds.blue, but it’s open source, so you can also run it yourself.
This is just a backend/API though, so how do you use it in practice, what about the UI? I built the UI as part of Skythread, because it already had code for authentication, making requests and rendering posts and threads in JS, so there was no point duplicating that elsewhere. (Skythread is also open source, and at least right now you can open it directly from the downloaded repo, without even running any builds or dev servers, since it’s just plain HTML & JS with almost no dependencies.)
You can find it in a somewhat new, separate section of Skythread that requires authentication and includes a few different personal tools:
- Posting statistics, which scan your home timeline or selected user’s profiles and shows you statistics on who posts how much per day on average
- Like statistics, for finding people who have liked your posts the most and whose posts you have liked the most recently
- Timeline search, which lets you fetch several days worth of posts from the “Following” feed and search for that thing you’ve briefly seen two days ago
- and Archive search, which is the UI for Lycan
You can find links to all of these in the popup menu accessed through the avatar button in the top-left corner. This is probably not the final version of this navigation, I need to rethink it all, but this will have to do for now.
Authentication is also required because I made Lycan require service auth (atproto-proxy) of a specific user, so that you can only import and search your own likes - because I thought that even though it’s technically public data, it would be kinda creepy if you could easily search for any keywords in someone else’s likes 😬 I don’t wanna build a Torment Nexus accidentally.
I haven’t added OAuth yet to Skythread - sorry, I know, I want to look into that in near future. Note, the access token isn’t stored on the server - it’s only used locally on the client, to make calls to your PDS for the threads and statistics, and in case of Lycan, the requests are also made to your PDS, which “proxies” them (same as with client<->PDS<->AppView normally) to the Lycan server, passing it a different kind of token, which only proves that it’s you making the request, but can’t be used to act on your behalf.
So once you log in, press “Start import” and then come back in a bit to see if it’s finished. Then use the search field and the category selectors to search in your archive. The search currently matches full words, with no modifications like plurals, with support for both phrases in quotes and exclusions with “-” (so e.g. "swift released" -taylor should work). I have a few more things on the todo list, e.g. searching in image alt text or searching in multiple categories at the same time.
From the implementation side, the search is rather “bruteforce” at the moment, i.e. there’s no separate search server like Elastic or a Postgres full text index, it just does a plain old LIKE search (or more precisely, ~* with a regexp for the word boundaries support) across all posts in your archive. I’m hoping this will be enough for now, with a fairly low number of users and the dataset mostly fitting in memory (it’s something on the order of ~50 MB per user on average).
I’m aware it’s likely that this feature will be “sherlocked” at some point by Bluesky when they implement this in their app (hopefully!) - but it’s not like I’m making any money on this 🙃
BTW, speaking about Skythread, I’ve been slowly approaching the point (or might be well past it, tbh) where “rawdogging the DOM” stops being a reasonable approach to building the app, so I’ve grudgingly started looking into migrating to some kind of proper modern UI framework, while still keeping things as simple and not frameworky as humanly possible. Looks like Svelte is winning so far (I started going through the official tutorial).
Apart from that, some other things I have planned next:
- better navigation
- OAuth support with limited permissions
- mobile layout (yes, I remember 😛)
- more configuration, including things like overriding the default AppView
- quick reply panel inside the thread
- using Constellation for “Hidden replies”, so I can load them more than a month back
To not miss any updates, please lycansubscribe (I’m really sorry…)
