Update To Search API

[-] DubTeeDub | 66 points | Jan 29 2018 19:37:47

Is there a way for us to search for posts on a subreddit within a certain date like we used to with the search functions?

This was hugely beneficial for us during our yearly Best Of awards so users could easily see the top posts every month

[-] priviReddit | 4 points | Jan 29 2018 19:42:40

you will still be able to search within last month or last year but not within a specific start and end date.

[-] DubTeeDub | 73 points | Jan 29 2018 19:43:22

that is really unfortunate

Is there any intention of bringing back those search queries in the future or is that going to be impossible?

[-] priviReddit | -28 points | Jan 29 2018 19:48:33

Currently it is not on our roadmap.

edit: *not

[-] DubTeeDub | 89 points | Jan 29 2018 22:33:25

This is a pretty big deal

Being able to search up top posts from past years was a really nice feature

Particularly since reddit continues to grow the top posts of all time are just going to end up mostly being from the last few months

This is also making it a lot harder to stop serial reposts and karmawhores who go back to find top posts from a couple years back and share them again as original content

I don't get why you all would remove this functionality that was part of reddit for a decade

[-] Watchful1 | 14 points | Jan 29 2018 23:02:28

There's probably a pretty good argument for removing it. Indexing stuff like that is likely really expensive. Just keeping the top 1000 posts in the all time, year, month, etc is comparatively trivial.

But it does suck for everyone who is using it. I wish they had taken user feedback when the start planning the new search and removing this rather than everyone finding out how important it is now when it's likely far too late to change it.

[-] svn | 28 points | Jan 30 2018 10:43:52

No money for search.

Money for a complete rewrite of the site that no one wants.

[-] CelineHagbard | 14 points | Jan 30 2018 10:53:59

Oh I'm sure their corporate clients and "partners" have access to these features. There is money in search, and they're charging for it.

[-] Watchful1 | 3 points | Jan 30 2018 18:15:30

It's definetly not so simple as throwing money at the problem. There are big complexity costs to the engineering decisions surrounding an entire new search system. Writing a system that can easily and quickly return arbitrary results based on a time period is a very different system than one that doesn't need to do that. You need to design the whole thing differently from the ground up, and maintaining it or making changes in the future gets more complicated.

It could definetly be done, but likely not this late in the process. And as annoyed as we are about it, we are a tiny minority of people who will be using the new system.

[-] svn | 2 points | Jan 30 2018 19:06:55

They haven't written a thing... they use external services

[-] Watchful1 | 1 points | Jan 30 2018 19:11:12

They certainly use existing software, but integrating it with the existing databases and configuring it how they want it to work is likely the full time work of a half dozen engineers. I highly doubt they use some externally hosted service. It's just not feasible on the scale reddit works at.

[-] svn | 6 points | Jan 30 2018 19:54:50

...they were using CloudSearch

[-] Deimorz | 2 points | Jan 30 2018 21:04:45

The new search is a partnership with Lucidworks Fusion: http://www.marketwired.com/press-release/lucidworks-boost-search-reddits-massive-online-community-270-million-monthly-users-2232886.htm

(mention for /u/svn)

[-] Watchful1 | 2 points | Jan 30 2018 21:17:13

I guess you would know, but doesn't that mean that it's still reddit's engineers installing it on reddit's servers? I really doubt that time period based searching is simply a matter of clicking a button to turn it on and another button to bring another hundred servers online to handle to extra processing.

[-] Deimorz | 3 points | Jan 30 2018 21:22:18

I don't know any details about how it works or how it's integrated with reddit. In general though, time-based search should not be complicated, the time that something was posted is basically a single number that never changes and can be easily and efficiently indexed. Many, many search systems require the ability to search by specific time ranges and it's surely a feature that's readily available.

[-] Watchful1 | 3 points | Jan 30 2018 21:26:43

Don't suppose you would be willing to bug some people to find out why /u/priviReddit says it's not on the roadmap? I'm mostly posting out of frustration since IMO it's a very important feature for the bot and scripting community and it's just being summarily dropped.

[-] Deimorz | 3 points | Jan 30 2018 21:58:37

Oh, I stopped working at reddit over a year ago now, so I don't really have any more access to private/internal info than you do.

[-] kemitche | 11 points | Jan 31 2018 00:57:01

Timestamp based search has only been around since early/mid 2012, so about 6 years. I'm still sad to see it go though - it's one of things I'm proudest to have added to reddit :)

[-] DubTeeDub | 8 points | Jan 31 2018 01:02:05

Thanks for putting it together

It was a really cool feature

[-] AltLogin202 | 7 points | Jan 30 2018 13:50:22

&rt; This is also making it a lot harder to stop serial reposts and karmawhores who go back to find top posts from a couple years back and share them again as original content

As is often the case in these threads, you’ve answered your own question.

The most important metric for ad-driven websites is engagement. When functionality that makes interactions more efficient is removed or when users are generating more content (reposts) - even when that content isn’t particularly high quality or original - engagement goes up. When engagement goes up, so does ad revenue.

It really makes one wonder: if the leadership’s attitude towards volunteer moderators - the group most responsible for driving growth - is so piss poor (dismissive, snark, or insulting), just how badly do they treat their paid employees?

[-] DubTeeDub | 10 points | Jan 30 2018 14:20:34

Edit: my account is now suspended

Edit 2: im back

&rt;just how badly do they treat their paid employees?

Its actually been reported that since spez came back they restarted their drinking culture in the office and have had to deal with a shitload of sexual harrassment, so yeah about what I expect out of them

https://techcrunch.com/2016/07/21/reddit-is-still-in-turmoil/

&rt;One individual speculated that the reemergence of the company’s drinking culture was to blame for the uncomfortable environment. Under Pao’s reign, Reddit tried to eradicate the bro-like amount of alcohol consumption at the office, but that went right out the window following Pao’s departure in July 2015.

&rt;“During all the leadership regimes, there were multiple incidents where employees would drink too much and end up in embarrassing and inappropriate situations,” a source explained. “There were multiple sexual harassment complaints from both female and male employees against female and male employees stemming from incidents that generally happened when employees were drinking.”

&rt;Several employees fended off uncomfortable comments from users and management alike, sources claimed. “Management is terrible, a complete reflection of what the site is like,” one source said.

Also in case your wondering why all the women and minorities leave reddit

&rt;So why can’t Reddit seem to hang on to its employees — particularly women and people of color? The same source who described management issues told us “working at Reddit is kind of like having an abusive boyfriend.”

&rt;&rt;You care deeply for it. You believe in it. You want to make it better. You think you just might be the person that can make that happen. Then one day you realize how hard you have worked to make positive changes only to have it constantly chip away at your sense of self and continue the same toxic behavior no matter what you do.

&rt;&rt;That toxic behavior, including the disturbing content and harassment commonly found on reddit, targets women on the site and within the company at a far greater rate than men. Eventually you have to decide if you want to be a part of that. Is it healthy to continue working there? Many of us have had to seek therapy for PTSD since leaving. I don’t think anyone realizes or acknowledges the emotional damage that can occur from an environment like that.

&rt;&rt;It’s not surprising to me when women leave.

[-] douko | 21 points | Jan 30 2018 02:42:18

So, for example, all those fun, good looking infographics about sub activity, etc. during specific periods of time - gone?

Why does it seem like Reddit is intent on removing features for no real good reason?

[-] KimPeek | 12 points | Jan 30 2018 03:10:48

Jesus, they gave you profiles. What else do you want? /s

[-] douko | 9 points | Jan 30 2018 03:18:35

Chat is the most useful and def. necessary thing, so I guess I should just be grateful

🤢

[-] onlyforthisair | 6 points | Jan 31 2018 12:21:16

Please put it on the roadmap.

[-] MuchBetterTitle | 0 points | Jul 16 2018 01:56:04

You need to put this on the roadmap.

Make it a thing: Reddit Time Machine.

It is your duty for history's sake.

[-] Bratmon | 41 points | Jan 29 2018 21:43:14

Can I get a heads up on what useful feature you're going to remove next?

I assume this is a gradual depreciation; you're going to remove a useful feature every couple months until Reddit is just a blank page.

This has the added advantage that the UI designers will finally have the amount of padding they seem to want.

[-] ketralnis | 17 points | Jan 29 2018 21:56:24

Next Monday I was planning to remove all of the vowels, but I'm open to other ideas

*dt: spllng

[-] douko | 18 points | Jan 30 2018 02:42:55

Hahaha, v. funny, please stop removing features that aren't replaced/improved upon

[-] Tensuke | 4 points | Jan 30 2018 13:01:10

I've been to the moldy basement, I don't need that again!

[-] auxiliary-character | 2 points | Jan 30 2018 09:28:17

\&rt;:(

[-] r_notfound | 2 points | Jan 30 2018 14:55:46

I have a suggestion for something you can remove: stemming. Or at least give us the stemming:0 or stemming:no capability. Unfortunately, I have no expectation this will ever happen, despite the fact that it makes search results not match search terms, and has done so for years.

[-] ketralnis | 3 points | Jan 30 2018 19:14:44

Generally you'll get exact matches first before any stemmed matches. Also I'm pretty sure you can do exact-term matching (not stemmed) by quoting the term. For example askreddit "running" (to not match "runs")

[-] dronpes | 20 points | Jan 30 2018 06:37:06

Just wanted to add another voice regarding the removal of the timestamp functionality with this update.

The back-end restructuring of the Reddit search solutions have been awesome to watch. But this appears to be a significant strategic gap.

Reddit's wealth of information is tightly coupled to the day/week/month's context for many communities, and being able to retrieve content from certain periods is immensely useful to us.

In our dream scenario, we'd actually love to see increased options on this front - even for end users.

Reddit is home to a quarter million Pokemon GO enthusiasts (and arguably the veritable center of the entire game's global community) over on /r/TheSilphRoad, and we've developed a fantastic culture of analysis and research in our community.

But information changes by the week on our boards. Being able to search specific keywords in the context of specific time periods would be a game changer for us. (Something we've had to do via API previously, meaning often only the mod team or our most dedicated researchers were able to do so.)

Please consider adding the ability to use timestamps in a future iteration, and if possible, consider allowing the average Redditor to tap into the historical treasure trove that is timestamp-contextualized searching.

We'd be happy to chat more about this and answer any questions or illustrate use cases further.

Edit: Just took a look at the r/changelog announcement post about this and ... 3 of the 5 top comments are requests for greater control over the date ranges of the search parameters. I hope this helps illustrate the relevancy and utility of this. :)

[-] CelineHagbard | 14 points | Jan 30 2018 11:02:21

Adding my voice that this is an awful decision for the many devs who have spent our own free time developing the tools to make your site usuable for moderators and users.

Is there any way to add the ability to filter by id range? It would then be easy enough for us to compute/calculate rough timestamps based on a timestamp/id lookup table on our end, and then still be able to filter.

[-] midnight-souls | 7 points | Jan 29 2018 22:36:54

So I might as well delete the bot I spent over a year writing. Fuck reddit.

[-] CelineHagbard | 5 points | Jan 30 2018 10:55:41

/r/pushshift

There might be hope, depending on what you were doing with it.

[-] aqouta | 6 points | Jan 31 2018 20:08:21

Is there actually any way to search for posts older than 1 year and further down than the 1k most recent posts anymore? It's really frustrating to navigate some subs that have regularly releases but timeless content.

[-] Sophira | 4 points | Feb 01 2018 01:44:59

That's a huge shame - I use this feature all the time. Reddit's use as a historical artifact is amazing. Being able to use timestamp: to search by specific dates is great when searching for what people were posting in response to specific events.

Also, you mention searching by "last month" and "last year" - how about the other "last X" options currently available?

[-] 13steinj | 3 points | Feb 03 2018 10:30:22

Adding yet another voice as to how not only is this an awful decision due to the wide variety of applications, especially data analysis in my opinion, however, also, I just plain don't understand. How can pushshift, a third party solution, which reflects not only post data but also comment data, for the use of searching, provide date range querying, but not reddit itself?

[-] Norci | 2 points | Jan 31 2018 13:27:07

Why was this feature removed?

[-] PM_ME_HAIRLESS_CATS | 2 points | Feb 01 2018 05:02:38

That's a pretty big rollback on a feature that would appear commonplace. Perhaps you should evaluate whether or not removing functionality is a value add when you keep adding features that are utterly pointless.

[-] uberafc | 1 points | Mar 23 2018 12:20:25

So will posts older than a year not show up for us regular users?

[-] priviReddit | 1 points | Mar 23 2018 14:51:32

They will be

[-] kungming2 | 1 points | Apr 01 2018 18:02:22

They'll still show up - there's just no way to construct a search query that say, returns results from three to two years ago.

[-] _korbendallas_ | 43 points | Jan 29 2018 19:30:18

"upgrade"

So searching by timestamp no longer works on api either?

[-] ketralnis | 5 points | Jan 29 2018 19:33:56

That is correct

[-] _korbendallas_ | 32 points | Jan 29 2018 19:45:31

That breaks a ton of moderating tools and other bots I use. Any replacement or do I just use "after" until I get what I'm looking for?

I do appreciate the efforts you guys make, it just seems like a pattern where "new" features solve imagined or real problems, but drop functionality that the "old" feature implemented well (or well enough).

[-] ketralnis | 8 points | Jan 29 2018 19:59:09

Can you be more specific about the use-case you're concerned about? How do these moderation tools use search? What tool is it and how does it work?

[-] D0cR3d | 41 points | Jan 29 2018 21:17:52

/r/DestinyTheGame has our weekly This Week In r/DTG History and I use this very timestamp method to find posts made exactly 1 year ago during the same timestamp. With the depreciation of this search capability would mean it'd be impossible for us to have this same post because there'd be no way to easily filter besides pulling all posts within the last 1 year which would be limited to last 1000 anyways, and do filtering.

I would really appreciate the ability to access this same information.

[-] GoldenSights | 31 points | Jan 29 2018 21:57:50

I have an entire program called Timesearch based on this feature. Over the past two years or so (the repo is new because I migrated the project) I've had several dozen community members and moderators benefit from the ability to collect a subreddit's history this way. I could get several testimonies if I asked.

Removing this endpoint would be the nail in the coffin for my interest in reddit programming, personally.

[-] beebacked | 6 points | Mar 22 2018 12:20:56

As someone who has done quite a bit of research around reddit communities, your tool is incredibly important to me and the work I do.

Getting rid of timestamp search is an archival disaster.

[-] ri0tnrrd | 2 points | Jan 30 2018 17:23:45

Was about to PM you but seeing as how this is your most recent comment I'll just mention it here. It seems that (at least for me) while running the timesearch for subreddits works stellar, running it for users keeps giving the following error(s). I've tested it via your timesearch program, and via the most recently updated Prawtimestamps on your reddit dir for github. For the timesearch version I get the following traceback:

binarybitch@leda:~/timesearch$ python3.6 timesearch.py timesearch -u goldensights
New database ./users/@goldensights/@goldensights.db
Traceback (most recent call last):
  File "timesearch.py", line 11, in &lt;module&rt;
    status_code = timesearch.main(sys.argv[1:])
  File "/home/binarybitch/timesearch/timesearch/__init__.py", line 425, in main
    args.func(args)
  File "/home/binarybitch/timesearch/timesearch/__init__.py", line 329, in timesearch_gateway
    timesearch.timesearch_argparse(args)
  File "/home/binarybitch/timesearch/timesearch/timesearch.py", line 151, in timesearch_argparse
    interval=common.int_none(args.interval),
  File "/home/binarybitch/timesearch/timesearch/timesearch.py", line 79, in timesearch
    new_count = database.insert(chunk)['new_submissions']
  File "/home/binarybitch/timesearch/timesearch/tsdb.py", line 208, in insert
    common.log.debug('Trying to insert %d objects.', len(objects))
AttributeError: module 'timesearch.common' has no attribute 'log'

Ok I just went in and removed all instances of log.common blah blah blah from tsdb.py and it's running for user just fine now

And yet when trying via Prawtimestamps I get the following:

binarybitch@leda:~/Prawtimestamps$ python3.6 timesearch.py timesearch -u ri0tnrrd
New database ./users/@ri0tnrrd/@ri0tnrrd.db
Traceback (most recent call last):
  File "timesearch.py", line 4, in &lt;module&rt;
    status_code = timesearch.main(sys.argv[1:])
  File "/home/binarybitch/Prawtimestamps/timesearch/__init__.py", line 425, in main
    args.func(args)
  File "/home/binarybitch/Prawtimestamps/timesearch/__init__.py", line 329, in timesearch_gateway
    timesearch.timesearch_argparse(args)
  File "/home/binarybitch/Prawtimestamps/timesearch/timesearch.py", line 146, in timesearch_argparse
    interval=common.int_none(args.interval),
  File "/home/binarybitch/Prawtimestamps/timesearch/timesearch.py", line 72, in timesearch
    for chunk in submissions:
  File "/home/binarybitch/Prawtimestamps/timesearch/common.py", line 62, in generator_chunker
    for item in generator:
  File "/usr/local/lib/python3.6/dist-packages/praw/models/reddit/subreddit.py", line 451, in submissions
    sort='new', syntax='cloudsearch'):
  File "/usr/local/lib/python3.6/dist-packages/praw/models/listing/generator.py", line 52, in __next__
    self._next_batch()
  File "/usr/local/lib/python3.6/dist-packages/praw/models/listing/generator.py", line 62, in _next_batch
    self._listing = self._reddit.get(self.url, params=self.params)
  File "/usr/local/lib/python3.6/dist-packages/praw/reddit.py", line 367, in get
    data = self.request('GET', path, params=params)
  File "/usr/local/lib/python3.6/dist-packages/praw/reddit.py", line 472, in request
    params=params)
  File "/usr/local/lib/python3.6/dist-packages/prawcore/sessions.py", line 181, in request
    params=params, url=url)
  File "/usr/local/lib/python3.6/dist-packages/prawcore/sessions.py", line 124, in _request_with_retries
    retries, saved_exception, url)
  File "/usr/local/lib/python3.6/dist-packages/prawcore/sessions.py", line 90, in _do_retry
    params=params, url=url, retries=retries - 1)
  File "/usr/local/lib/python3.6/dist-packages/prawcore/sessions.py", line 124, in _request_with_retries
    retries, saved_exception, url)
  File "/usr/local/lib/python3.6/dist-packages/prawcore/sessions.py", line 90, in _do_retry
    params=params, url=url, retries=retries - 1)
  File "/usr/local/lib/python3.6/dist-packages/prawcore/sessions.py", line 126, in _request_with_retries
    raise self.STATUS_EXCEPTIONS[response.status_code](response)
prawcore.exceptions.ServerError: received 503 HTTP response

[-] GoldenSights | 2 points | Jan 30 2018 18:38:01

From now on, you can ignore the reddit/Prawtimestamps repository, I moved timesearch to its own repo which is where all new updates go. This is mainly so you can simply git clone and git pull to get updates instead of having to fiddle with individual files.

The 503 error means the server was temporarily unavailable so that's no big deal. Just try again soon.

I'm not sure why you're having the "no attribute log" error, it's definitely there. Sounds like your system might be importing an old version of the files. Can you try recycling all the timesearch code and downloading clean from the repository?

[-] ri0tnrrd | 1 points | Jan 31 2018 19:02:34

Weird - I'll go double check and ensure that I'm using the most recent PRAW version, and will scrap the Prawtimestamps thanks for letting me know.

[-] _korbendallas_ | 24 points | Jan 29 2018 21:16:43

/u/RepostSentinel is a bot I manage that scans for reposts in a subreddit and currently moderates two former defaults and a dozen or so subs over 100k subscribers. Source code here. The way it imports old submissions from a sub uses the timestamp search week by week.
I have a script that compiles best of statistics for top comments of the year. Older version here.
I'm working on a bot that compiles albums of the top 10 comments from photoshopbattles threads per week/month and creates an imgur album to post to a meta sub.

Other scripts are more one time use and probably don't need the timestamp search.

[-] D0cR3d | 3 points | Jan 30 2018 19:05:57

&rt; /u/RepostSentinel

I think we could get around this by using the Database that TheSentinelBot uses and have it log the post data to that, and then just search based on the post timestamp in our local Database and we can just grab the URL from there. If we don't already store the URL for that we can add that, but pretty sure we do.

[-] Watchful1 | 22 points | Jan 29 2018 22:58:28

This is a really big deal. As far as I know, timestamp based searching has been the only way to get submissions that are past the 1000 post limit in the various listings. Anything that tries uses the praw submissions function that takes advantage of this will break.

[-] daily_digest | 15 points | Jan 30 2018 00:50:56

Not a moderating tool, but I have a site that allows people to get post from the last 24 hours for subreddits of their choice. Now I’ll have to make multiple calls to iterate through the last posts until I get to the previous 24 hours which is a significant increase in calls. Previously, through time based searches, I could limit the number of calls I needed to make. Maybe the cost of indexing should be weighted against the increase in network traffic?

[-] MystK | 5 points | Mar 10 2018 07:18:38

Howdy, I just saw this thread and didn't realize that the deprecation of the timestamp search was what was breaking the bot.

I am the developer of the bot that runs /u/changemyview.

We have weekly, monthly, yearly, and all time boards of points awarded. We used the timestamp search to grab the history of comments to parse these boards. Without the timestamp search, we aren't able to accurately populate these boards.

[-] rasherdk | 3 points | Apr 05 2018 11:48:34

So you removed a feature even without figuring out first if people were actually using it for important shit? And then when they tell you, you close your ears and pretend you heard nothing. Prime reddit right here.

[-] midnight-souls | 15 points | Jan 29 2018 22:33:46

So if I want to find posts from 2 years ago, what do I do?

[-] douko | 13 points | Jan 30 2018 16:35:19

"You fuck yourself, bud." - This admin, probably

[-] douko | 8 points | Jan 30 2018 02:43:58

So, not an upgrade, then?

[-] inspiredby | 4 points | Apr 05 2018 01:14:51

Please don't remove this, it's really useful for research.

[-] DubTeeDub | -6 points | Jan 29 2018 19:44:44

woomp woomp

[-] MajorParadox | 27 points | Jan 29 2018 19:30:19

Does this have anything to do with the "show legacy search page" preference? I still prefer the old search layout because it works like a filter. The new layout just makes me feel like I'm not on reddit anymore.

[-] ketralnis | 23 points | Jan 29 2018 19:31:03

It's unrelated, that only controls the rendering. I can't pretend that we'll support it forever but it's not being affected here

[-] MajorParadox | 5 points | Jan 29 2018 19:31:26

Ah OK, cool!

[-] Sophira | 1 points | Feb 01 2018 01:49:33

It'll affect the ability to use the syntax=cloudsearch URL parameter though, right?

[-] ketralnis | 2 points | Feb 01 2018 01:50:08

Yes

[-] 9Ghillie | 26 points | Jan 29 2018 19:33:21

Any info on improving/fixing/restoring the search functionalities? Searching by flair still seems to be broken and the timestamp based search feature was removed completely, which is greatly missed.

[-] priviReddit | 9 points | Jan 29 2018 19:40:10

Can you elaborate on what you mean by searching by flair being broken?

[-] 9Ghillie | 15 points | Jan 29 2018 19:45:01

In the case of my test, I searched for flair:potm in /r/itookapicture. Photo of the Month contest winners are flaired PotM [Month] [Year] and so far there's 7 of them, search only gives 5 results.

[-] ketralnis | 17 points | Jan 29 2018 19:51:50

Huh, 5 results but not 0. I bet we're not updating the search index on flair changes. I'll add it to the bug list

[-] 9Ghillie | 9 points | Jan 29 2018 19:54:16

Yeah, just some results seem to be missing.

[-] MajorParadox | 12 points | Jan 29 2018 19:51:35

I don't know about the API, but searching by flair class went away the last time search was updated. No way to search for a specific category that can have different text now.

[-] ketralnis | 4 points | Jan 29 2018 19:52:26

Can you show me an example search that should work but doesn't?

[-] MajorParadox | 10 points | Jan 29 2018 19:54:44

This used to be our "Mods' Choice" filter search, but it no longer works.

[-] ketralnis | 7 points | Jan 29 2018 19:57:18

Thanks! I'll see where that went wrong

[-] MajorParadox | 9 points | Jan 29 2018 20:00:22

Thanks! I dug up the last conversation I had about it here and it sounded like the functionality for css_classes was removed.

[-] Aiwayume | 3 points | Jan 29 2018 20:09:57

I would LOVE for this to come back, not sure if it is something that /u/ketralnis can take back as a feature request, but if not I understand (subs I mod used this to help users a lot, and when that functionality was removed, we ended up with some broken functionality)

[-] antiproton | 7 points | Jan 29 2018 20:04:41

Not for nothing, but it feels REALLY bad when these issues only get surfaced when a thread like this rolls around. The last search update was god only knows how long ago, and clearly the devs didn't know.

We need a way to submit issues and track the progress. Make it complicated, make it require 4+years old account, make submission only work on Tuesday afternoon... whatever it takes.

I get that it would be a bear to moderate and manage, but you have to ask yourself - how many more things could be logged and improved that you didn't even know where an issue in the first place?

[-] priviReddit | 2 points | Jan 29 2018 21:03:28

Thanks for the feedback. In the short-term, feel free to surface bugs on this thread or on r/bugs. If you encounter an issue in the future please reach out at contact@reddit.com or /r/reddit.com modmail and we'll take a look.

[-] Deimorz | 6 points | Jan 30 2018 05:31:40

I've seen a number of search bugs reported over the last few months in /r/bugs. Some of them were reported multiple times, and some of them have been commented about again in this thread.

As far as I saw, none of those posts received a response, and none of the bugs were addressed. Is someone going to start actually paying attention to /r/bugs?

[-] throwaway_the_fourth | 0 points | Jan 30 2018 02:33:02

The problem with /r/bugs is that it's absolutely flooded with posts that are decidedly not bugs, like "I can't log into my account." Reddit admins already seem to have enough trouble sifting through the feedback they receive, so I doubt that most posts on /r/bugs are ever read by an admin.

[-] Deimorz | 5 points | Jan 30 2018 06:10:22

That's really not much of an issue. Even with all the mistaken posts (and the insect photos), it still usually only gets about 10 submissions per day. It only takes seconds to skim through it quickly.

[-] DubTeeDub | 1 points | Jan 29 2018 19:44:24

search by flair seems to be working for me

https://www.reddit.com/r/BlackPeopleTwitter/search?q=flair%3AQuality%2BPost&restrict_sr=on&sort=new&t=all

[-] Jakeable | 20 points | Jan 29 2018 19:31:30

Something I've noticed with the new search is that certain characters don't work. Queries with question marks (example) don't seem to work. Is this an intentional design choice?

I've also noticed that the site parameter doesn't seem to work as expected anymore. For example this search for site:yahoo.com also returns results for other sites that contain yahoo.com in the url.

I don't think these queries are anything fancy or special, I just want them to work properly :(

[-] ketralnis | 12 points | Jan 29 2018 19:36:46

&rt; Queries with question marks (example) don't seem to work. Is this an intentional design choice?

That URL looks like https://www.reddit.com/r/politics/search?q=What%27s+behind+rich+people+pretending+to+be+self-made?&restrict_sr=on but in HTTP URLs, ? is a special character. You'll need to escape the ? as %3f like you would in any URL.

Unless I'm misunderstanding the problem you're having

&rt; this search for site:yahoo.com also returns results for other sites that contain yahoo.com in the url

Hmm yeah that looks like a bug. I'll take a look

[-] Jakeable | 18 points | Jan 29 2018 19:38:11

I made that search URL using the search bar in the sidebar of a subreddit. I understand escaping it if this was an API call, but I think if searching with a question mark from the front end it should be auto escaped.

&rt;Hmm yeah that looks like a bug. I'll take a look

Thanks, I appreciate it.

[-] ketralnis | 13 points | Jan 29 2018 19:39:31

Ah gotcha, so it could be an issue on either side (web frontend or query backend). I'll take a look at both then

[-] therealandytuba | 4 points | Jan 29 2018 19:39:33

Whether API or HTML request, that's still a URL and question marks need escaping in URLs because they're special characters. Lemme look at this a little to see if something needs fixing (or maybe my understanding of things)..

edit: Yes, the reddit sidebar search should url-encode question marks correctly: https://i.redd.it/l6o7a4vo82d01.gif

[-] ketralnis | 8 points | Jan 29 2018 19:42:24

Yeah, I think what /u/Jakeable means is that they didn't type that URL, they got that URL by using our actual HTML form element like a regular human person would do

[-] Jakeable | 5 points | Jan 29 2018 19:43:07

That is correct, sorry for the miscommunication.

[-] therealandytuba | 2 points | Jan 29 2018 19:45:23

As a regular human, I repeated what u/Jakeable was describing--typing a string with a questin mark into the right sidebar search box-- and r2 seems to url-encode the ? correctly: https://i.redd.it/l6o7a4vo82d01.gif

Jakeable, is that the method you used? maybe mobile web or redesign has the bug? Could you try giving it a go and see if you can figure out the repro steps for the URL you shared above?

[-] Jakeable | 6 points | Jan 29 2018 20:02:37

Yeah I just tested it again and still encountered this error.

Safari

Reddit for iOS

I tested it on these browsers and still encountered the error:

Safari 11.0.2 (logged in and logged out, no extensions either time)
Chrome (logged out, all extensions disabled)
Reddit for iOS v4.2.0.301113 (logged in and anonymous mode)

[-] therealandytuba | 4 points | Jan 29 2018 20:16:35

I see from your Safari gif that the URL is encoded correctly (note the %3F in the address bar), so I imagine there's a different error happening right now. Maybe the search boxes really are overloaded at the moment.

[-] ketralnis | 6 points | Jan 29 2018 20:19:31

I think I'm just wrong about the original URL and the problem is actually with search

[-] therealandytuba | 5 points | Jan 29 2018 20:28:59

Yeah, does seem like a problem with search itself, especially since Jakeable and I are both seeing error result with correctly-encoded query params.

[-] Jakeable | 1 points | Jan 29 2018 20:18:40

I did try searching “question” or “test” before and after each “question?” search, and those tests didn’t fail. This issue has also been occurring for several months now.

[-] therealandytuba | 5 points | Jan 29 2018 20:30:26

Thanks for QAing with different variants -- same text, no question mark; different text, no questoin mark! Sounds like it's on u/ketralnis' radar now and hopefully he'll sort it out.

[-] Jakeable | 1 points | Jan 29 2018 20:30:47

No problem! Thanks for looking into this.

[-] Jakeable | 4 points | Jan 29 2018 19:40:38

I understand that, but I don't think it's the best user experience if regular users (who might not understand or care about escape characters) have to escape a question mark to search something if they're using reddit's frontend.

[-] mavoti | 4 points | Jan 29 2018 22:56:11

/u/ketralnis

&rt; You'll need to escape the ? as %3f like you would in any URL

/u/therealandytuba

&rt; question marks need escaping in URLs because they're special characters

That’s not correct.

Inside the query component, the ? has no reserved meaning, so it can be used unescaped there.

[-] ketralnis | 2 points | Jan 31 2018 17:36:31

Huh, today I learned

[-] kungming2 | 20 points | Jan 29 2018 19:44:56

u/bboe, what does this deprecation mean for PRAW's submissions?

[-] bboe | 17 points | Jan 29 2018 23:11:21

It looks like submissions will have to be deprecated.

/u/priviReddit is anything in the works to enable the possibility to list all submissions for a given subreddit? Without the timestamp specific search it seems there is now no way via Reddit's API other than iterating through all ids to find all submissions for a subreddit.

Third party APIs like pushshift exist to provide this information, but there are people hesitant to rely on third parties for such information.

Finally, I just want to say thanks in advance for providing a heads up about the deprecation. I really appreciate the opportunity to make a proactive change to PRAW, rather than a reactive one.

[-] 13steinj | 1 points | Feb 03 2018 10:35:49

In theory it's possible to algorithmically predict posts' id ranges and distribution for a given subreddit over time, but this wouldn't be with any decent amount of certainty and furthermore would be inefficient because the maximum amount of posts that can be queried by id is 100.

[-] geitir | 2 points | Feb 14 2018 04:35:20

that would imply consistently measuring usage statistics for the entirety of reddit would it not? ie finding out that, for example, reddit is currently receiving 200 comments a second, 50 posts a second, 10 pm's a second, and then continuing to measure this?

[-] 13steinj | 2 points | Feb 14 2018 04:41:14

Something like that, yeah. Pushshift has live streaming capabilities with reddit on a small delay, so it's not impossible.

[-] geitir | 1 points | Feb 14 2018 04:44:23

oh cool, interesting

[-] xHaZxMaTx | 14 points | Jan 29 2018 20:26:33

Is there still not a way to search specifically for spoiler-marked posts like there is for searching for NSFW-marked posts, i.e. "spoiler:yes"?

Also: I noticed that it's no longer possible to search for specific time frames using time codes. This was suuuper useful for the annual Best Of nominations threads we'd make. Example here. Is there any plan to re-introduce this feature or a feature like it?

[-] ketralnis | 3 points | Jan 29 2018 20:34:35

&rt; search specifically for spoiler-marked posts

Not currently, and actually it seems a little weird to search specifically for spoilers (although I can imagine wanting to search while specifically excluding them). Can you talk more about what you have in mind there?

&rt; specific time frames [...] any plan to re-introduce this

No, not currently. I've heard mention of this "annual best of" use case a couple of times here in this thread. You can still limit searches to "past year" so I don't think I'm fully understanding what use-case is broken

[-] Rene_Z | 14 points | Jan 29 2018 21:26:47

&rt; I've heard mention of this "annual best of" use case a couple of times here in this thread. You can still limit searches to "past year" so I don't think I'm fully understanding what use-case is broken

"Past year" is a relative measure, you'll get different results if you click on it mid-December or mid-January. Also, a year later you won't be able to look at the top posts of two years ago, it'll just the results of the current year.

And more importantly, as can be seen in the linked post, there's a separate search for each month, which wouldn't be possible at all with the new search. And as subreddit activity varies throughout the year, the top posts of less active months would get buried further down in a search for the whole year.

[-] xHaZxMaTx | 1 points | Jan 29 2018 20:35:54

Thanks for the quick response!

&rt;Can you talk more about what you have in mind there?

Well, to be perfectly honest, it's not something that comes up often, and as a user it probably would never come up, but as a moderator it can be a useful tool.

[-] SirBuckeye | 13 points | Jan 29 2018 20:36:54

self:1 and self:yes still don't function at all since a change was made a few months ago. Any plans on restoring these operators?

[-] reseph | 5 points | Jan 29 2018 19:49:18

I believe I've been on the new search stack, and it has been generally broken for me. See:

https://www.reddit.com/r/bugs/comments/7fxpye/new_search_is_broken_site_and_self_do_not/

Am I doing something or is it just broken?

[-] ketralnis | 6 points | Jan 29 2018 19:54:09

Is that one still broken for you? We did change something related to this recently and it does work for me

[-] reseph | 8 points | Jan 29 2018 20:11:10

Still broken, aye. `self:yes` is still showing picture/Imgur results, `site` is still stuck from 6 months ago at the latest in my example (and there are recent examples from say 2 months ago to said domain).

[-] ketralnis | 3 points | Jan 29 2018 20:29:29

Weird, and thanks for double-checking me. Adding it to the list

[-] FiveYearsAgoOnReddit | 5 points | Feb 15 2018 00:20:01

This has meant the end of two quite popular subs, just for the record:

/r/FiveYearsAgoOnReddit
/r/TenYearsAgoOnReddit

which were fed by a bot using the cloudsearch timestamp feature.

Oh well. I'd wouldn't mind someone explaining why, as it's not March 15th yet.

[-] DiscoPanda84 | 4 points | Jan 29 2018 23:08:41

&rt; Besides some minor syntax differences, the most notable change is that searches by exact timestamp are no longer supported on the newer system.

...is that why all the guides and comments I've seen on things like finding my oldest post (either in a particular subreddit, or just on reddit as a whole) don't work at all and instead give me zero search results?

Edit: Is it really that odd for me to want to look at some of my older posts/comments? This is the first I'd seen any mention of anything that would explain why so many places would be suggesting a method that doesn't work at all...

[-] SometimesY | 4 points | Jan 31 2018 06:13:52

Hey thanks for slowly ruining our code base!

[-] Murica4Eva | 4 points | Feb 14 2018 02:36:19

This is awful. Why can't you make it easier to find old posts by time instead of harder? It seems like a obviously needed and easy to keep feature.

[-] Exaskryz | 3 points | Jan 29 2018 22:18:49

Yo, since we have a thread about searching, I just wanted to ask: Is there a way to limit your searches to subreddits you are subscribed to?

Just a day or two ago, I refreshed the front page. Silly me. Because wiħ the slight delay in it loading the refresh, a post caught my eye. It was an older post, and was removed from the front page as a result. I tried searching for keywords from the topic and trying to narrow it down to individual subreddits I thought it would be in, but to no avail.

[-] appropriate-username | 2 points | Jan 31 2018 21:40:43

Are you planning on maybe eventually adding search to modmail?

[-] 13steinj | 3 points | Apr 17 2018 08:50:35

Ha. We both know that /r/modmailbeta is in the gutter.

[-] appropriate-username | 1 points | Apr 17 2018 22:50:00

F

[-] irrational_function | 2 points | Feb 24 2018 17:31:30

Is there any way to do case-insensitive title searches with the new search stack?

You might say "use all lowercase", but sometimes exact-case gives matches that all-lowercase does not. For example, this title:jQuery search includes this result with jQuery/JavaScript in the title, but this title:jquery search does not.

It seems maybe like if the query term is only punctuation-separated in the title, not fully whitespace separated, then it needs to be exact case. I can't be sure of the exact rule.

A real case where this is a problem for bots is searching for username mentions in a title, as people may say "u\/username" in a title. A search for "title:privireddit" will match a title containing "priviReddit" or "privireddit" or "u\/privireddit", but not "u\/priviReddit". (A search for "title:priviReddit" will only match titles containing "priviReddit" and "u\/priviReddit", so that's no help.)

[-] Tsundere_Clegane | 2 points | Mar 25 2018 09:41:57

&rt;timestamps

Oh, that feature was actually quite nice, hopefully the staff implement some other way to deal with date based searches. Reading through archived threads definitely is something that date range search was really helpful for.

[-] spike77wbs | 2 points | Apr 05 2018 05:34:17

&rt;timestamp

So glad I wasted all that time helping you build content. Thanks.

[-] agatha361 | 2 points | Apr 06 2018 00:16:04

Did you just break RSS-based queries by switching those queries over?

See also:
https://www.reddit.com/r/bugs/comments/89wx7b/advanced_search_changed_semantics/

[-] 13steinj | 1 points | Apr 17 2018 08:37:46

Not all RSS based queries, just RSS based queries that use the cloudsearch syntax (and any queries that are shit on the new stack). RSS queries are done in the sams exact way as other API queries, just, well, rendered in atom/xml.

[-] _BindersFullOfWomen_ | 1 points | Jan 29 2018 19:46:27

As a squishy definitely human user of /r/totallynotrobots, I am glad to hear that the robots are losing their search abilities.

[-] priviReddit | 9 points | Jan 29 2018 19:50:00

Robots are scary !!

[-] MajorParadox | 4 points | Jan 29 2018 20:38:39

:(

[-] GangstaCuddles | 1 points | Jan 30 2018 22:33:12

Fuck you. Turn off the shitty myspace profiles.

[-] escortspoint | 1 points | Feb 19 2018 10:18:49

The best Independent Escort service in Bangalore. We Provide Hot and Sexy VIP Escort service in Bangalore round the clock. escorts in bangalore http://www.escortspoint.com/ http://bangalorecallgirlsclub.com/ http://escortsin-bangalore.com/ http://www.escortservice-delhi.com/ http://www.escortswala.com http://www.escortswala.in http://www.escortswala.co.in http://www.escortwala.co.in http://www.escortwala.in http://paqos.in/ http://www.akashbhatiseoexpert.online/

[-] w0ut | 1 points | Apr 09 2018 08:47:47

This is not good, in /r/art we use a bot to stop people from spamming multiple posts in an 48 hour time frame, and we use this call to poll the latest few posts and we check those against our database (python + praw):

submissions = reddit.subreddit('art').submissions(start = last_submission_date_timestamp)

The query returned exactly what was needed, without unnecessary traffic, and I don't even get from this what is replacing this.

I guess we'll just allow the spam and bloat reddit further with unwanted content, that seems to be the goal of this. At least the new search of all this spam will be fast?

[-] assertiveashwin | 1 points | Apr 10 2018 21:54:27

The same holier than thou attitude. We are the admins, so we do whatever the f*** we feel like. Kneeeeeel......

[-] 13steinj | 1 points | Apr 17 2018 08:49:15

Please notify the owners of bots and applications who this has caused an issue of.

It is clearly evident that you miscalculated the scale of applications that this would affect, and their intersection with the redditdev and changelog communities.

Given the massive amount of analytics you collect, I would think it relatively simple to query all OAuth app ids that have been hitting the /search endpoint and sending the developers an email from api@reddit.com and from /u/reddit. It should be noted it seems that given the comments on this thread, more than just cloudsearch has been affected, so narrowing it down to only cloudsearch users is not enough. Not to mention that email address was specifically noted to be for special api changes, and this is a large one.

Furthermore, it would be nice if you let them know of alternatives they now have, which are

hitting a third party API such as pushshift
hitting /api/info with consecutive ids and yielding results, filtering them as they yield

[-] rbevans | -1 points | Jan 29 2018 21:40:22

Oh nice, this will free up my spacebar now for other things.

What’s changing for regular users?

What’s changing for the robots?

When will this happen?