Talk:Wiki search engine upgrade project

There has been some discussion on the wiki's IRC chat channel about potential upgrades to the way the wiki does searches, and I want to make sure everyone is aware of the discussions. So, here they are:

July 18, 2012 17:52:01	MeliCat	heya 17:52:17	MeliCat	why can't i find https://ddowiki.com/page/Greater_Shout when i type 'Greater Shout" in the front page? 18:03:20	MeliCat	anyway... sorry but if the wiki gets harder and harder to find stuff it's going to become as useless as the Compendium. Please may this not happen 18:03:24		*** MeliCat quit (Read error: Connection reset by peer) juz_at_lar@Rizon-7FED8813.perm.iinet.net.au ...Snip... 20:43:11	Shoe|Baby	The answer to MeliCat's question is that he did not click the checkbox for search titles only... Doing this puts Greater Shout on the top of the list when typed into search. ...Snip... 20:47:16	Trekna	To be fair the average user would expect to be taken directly to a direct match ...Snip... 20:50:33	ShoeMaker	That's not the way the http://www.sphinxsearch.com/ that the wiki uses works... It is no different than any other wiki that uses this search engine... 20:51:23	ShoeMaker	Although, I suppose that I could see if there was a better search engine to use... That may pose a problem with the FireFox plugin that the wiki has too though... 20:53:44	Trekna	Its just... Odd that searching for greater shout is actually searching for greater and shout independently 20:54:11	Sjachvorastrix	Did you try putting it in quotes? 20:54:15	Trekna	Yes 20:54:38	Trekna	Doesn't make a difference 20:55:12	Sjachvorastrix	Ok... that is weird. 20:56:02	ShoeMaker	Give me a second.... 20:56:21	Sjachvorastrix	And yes, if I do a search for "Greater Shout", I expect any search engine to recognize that I intend it to find the complete phrase and not give me disjointed informaiton 20:56:23	Sjachvorastrix	information* 20:58:20	ShoeMaker	It is an option in the advance search, I am researching if there is a way for it to be checked by default or even better, a way for the search engine to recognize '...Search for this...' or "...Search for this.." and the operators AND/OR/XOR/NOT 20:58:23	Trekna	Using Marks query... It returns over 3000 results for greater and 64 for shout... Logically Greater Shout as a query could not possibly return more than 64 results.... Yet it returns 1000+ 20:58:27	Sjachvorastrix	Especially since there is no link on the front page to go to an advanced search page 20:58:54	Trekna	Marks = Melis....damn you autocorrect 20:59:45	ShoeMaker	The results page is the advanced search page at the bottom (unless you have an account, then you can specify some defaults). 21:00:30	Trekna	Save headaches by defaulting to match all not match some 21:01:31	Sjachvorastrix	Ok, now I find the link to go to the advanced search page on the home page... on the opposite side of the screen from the search box >.< 21:05:02	ShoeMaker	Wikipedia uses: https://www.mediawiki.org/wiki/Extension:MWSearch && https://www.mediawiki.org/wiki/Extension:OpenSearchXml && https://www.mediawiki.org/wiki/Extension:TitleKey 21:05:29	ShoeMaker	I'll talk to Xevo about us switching to that... This would allow for predictive search in the search on the home screen...

From: ShoeMaker 

To: Xevo --Hidden--

Sent: Wednesday, July 18, 2012 9:24 PM

Subject: Wiki search engine options... All of the ...Snip...s are unrelated bot responses and room modes and such... I was napping when the original question was posed. So, what do you think? I did some research on it, and it looks like MW1.17 uses a beta version of Sphinx, and 1.18 uses the first stable release... But I can't seem to find the change log on it to see what has been changed... Would be great if you could come into chat to figure out the best course of possible change for this. Do we upgrade to MW1.18 and keep using Sphinx and see what is new for a while, do we change over to the search setup that wikipedia uses (I do like their predictive search bar), or is there another alternative? If I don't answer right away, you can ask my bot (Kobold_Scribe) where I am (Where is Shoe?) and he should tell you a fairly accurate answer. I check the IRC about once every hour or so when I am awake, and slightly less often when I am sleeping. :p If I'm sleeping, I will see any discussion in the chat log of my bot and can send you another email if we miss each other. :D

July 19, 2012 From: Xevo --Hidden--

To: ShoeMaker 

Sent: Thursday, July 19, 2012 10:38 PM

Subject: Wiki search engine options... I have changed it so that it will "Search All" by default. Now when I search for Greater Shout, the expected page is the first result.

July 20, 2012 From: ShoeMaker 

To: Xevo --Hidden--

Sent: Friday, July 20, 2012 11:36 AM

Subject: Wiki search engine options... That is a start... I would still like to research the ability for predictive search like Wikipedia has... So when you start typing in "Gre"... It populates a dropdown list of everything that starts with "Gre" like "Great this", "Greater that", "Green stuffs", ...  I think that would be cool. :D Thanks, Shoe

From: Xevo --Hidden--

To: ShoeMaker 

Sent: Friday, July 20, 2012 12:03 PM

Subject: Wiki search engine options... I have a log file that contains every search that has been placed by users. Maybe it would be best to use previous searches as the input for the predictive search. I think that would work better than trying to predict a search off of the text in all the pages.

From: Xevo --Hidden--

To: ShoeMaker 

Sent: Friday, July 20, 2012 12:04 PM

Subject: Wiki search engine options... I suppose the one downside to this approach is that if somebody searches for something irrelevant or obscene, it would come back as a suggestion ;) Maybe I could only suggest keywords that have been searched for multiple times ?

From: ShoeMaker 

To: Xevo --Hidden--

Sent: Friday, July 20, 2012 12:24 PM

'Subject: Wiki search engine options... I think that Wikipedia has their predictive search set up to match page titles, and if your search doesn't match any title, hitting [Enter] takes you the search content / advanced search. Using your idea, maybe the search log could first be matched against a list of "restricted" words, the vulgar, obscene, and known spam... Then, whatever passes that test could be logged to a MediaWiki:Predictive_Search for review and modification my any admin to eliminate irrelevant/off-the-wall stuffs and parsed as the predictive matches? Thanks, Shoe

July 21, 2012 18:58:00	ShoeMaker	!8ball has x3v0 found a solution for the search engine yet? 18:58:01	Kobold_Scribe	Kobold is certain 18:58:07	ShoeMaker	Sweet. 18:58:14	ShoeMaker	Kobold says you have a solution! 18:58:31	ShoeMaker	My kobold magic 8ball that is. 18:59:38	x3v0	I'm not so sure about lucene 19:00:16	x3v0	it uses much more memory than sphinx search does 19:01:04	x3v0	i would probably have to run it on a separate server. i would be willing to do that if the search is a lot better, but i don't think it really is 19:01:48	ShoeMaker	Can the new stable version of Sphinx do the predictive search? 19:02:02	ShoeMaker	once we upgrade to 1.18 that is... 19:02:05	x3v0	i could probably get away with running it on the torwiki server though, that really isn't getting much traffic 19:02:07	ShoeMaker	I couldn't find release notes. 19:02:18	ShoeMaker	Mostly me and the spambots, no? 19:02:19	ShoeMaker	lol 19:02:22	x3v0	heh 19:02:29	x3v0	i think i have cut down on the bots 19:02:51	x3v0	I'm running the traffic through cloud flare now 19:02:55	x3v0	which block spam bots 19:03:01	ShoeMaker	Oh, nice. 19:03:18	x3v0	i may do that with ddo wiki, I'm trying it out with for now 19:03:24	x3v0	*with tor now 19:03:53	x3v0	i only saw 2 spammers last week, which is way less than normal 19:06:38	x3v0	id probably choose incapsula for ddowiki though 19:06:50	x3v0	i just think they seem more reputable. I'm not sure why 19:06:53		*** ShoeMaker is now known as Shoe|Baby 19:14:05	Trekna|Android	Good number of spam edits lately 19:17:35	x3v0	oh…awesome 19:17:37	x3v0	Search Box "As-You-Type" Suggestions 19:17:37	x3v0	Set the following options in LocalSettings.php after the above require_once line: 19:17:38	x3v0	$wgEnableMWSuggest - set to true to enable MediaWiki support for search box suggestions 19:17:39	x3v0	$wgEnableSphinxPrefixSearch - set to true to return suggestions by matching the query against the beginning of page titles 19:18:02	x3v0	the new version of SphinxSearch DOES support predictive search 19:22:19	x3v0	I'm going to upgrade to 1.18 this weekend 19:23:00	x3v0	I want to go to .19, but I think it would be too risky. We would be using a lot of extensions that were not fully tested in 1.19 19:28:56	Shoe|Baby	I was hoping so... 19:29:13	Shoe|Baby	I think it would be safe myself 19:30:18	Trekna|Android	Just happy it searches for all terms 19:30:44	Shoe|Baby	lol 19:30:56	Trekna|Android	instead of a mix

July 22, 2012 ...Snip... 09:59:34	ShoeMaker	Was the discussion about the search engine x3v0 and I had last night, and some comments on the spambots after.... For those interested. ...Snip... 10:01:19	ShoeMaker	Was the rest of the conversation. 10:03:11	ShoeMaker	I'll make a censored talk page about it on the wiki shortly. Snipping out the non-relevant stuffs. 10:11:49	x3v0	im working on that now 10:14:25	x3v0	shoe, are there some templates you can think of that make heavy use of certain extensions? im trying to find the best pages to look at to make sure nothing broke 10:14:51	x3v0	i have the new version installed an working on my virtual machine copy of ddowiki 10:18:37	ShoeMaker	The Named_xxx templates 10:19:11	ShoeMaker	They use all of the extensions directly or as a transclusion with all of the templates used in the templates on the item pages with templates. ...Snip... 10:21:05	x3v0	yea, that should be a good test ... More to come? ??? ShoeMaker (Contributions • ) 11:26, July 22, 2012 (EDT)

I assume that the recent change where it now lists the search box, several links about the search content (content pages, multimedia, help, etc.), and then all the places it could search (main, talk, user, user talk, etc.)... and THEN actually shows the links, is a result of any changes that're being made? Is there any way to, y'know, put all that back below the actual links that I'm just not seeing, or am I stuck having to scroll past all this crap I don't care about to get to where I might be able to find what I'm actually searching for? LrdSlvrhnd (Contributions • Message) 15:52, July 23, 2012 (EDT)


 * That is part of the overall wiki upgrade from 1.17 to 1.19 and we are working to make it look better and be more user friendly...

ShoeMaker (Contributions • ) 16:31, July 23, 2012 (EDT)


 * This looks good but I don't like images showing up on the search page by default. My 2cp. Book O Dragons (Contributions &bull; Message) 16:42, July 31, 2012 (EDT)


 * You could try going to your search preferences page and put a check in all the namespace boxes that you want to be searched (obviously leaving out file) and your results would be more refined for you by default... You can also click on the advanced search link if you wanted to add some of those namespaces back in for just one search.

ShoeMaker (Contributions • ) 09:08, August 1, 2012 (EDT)