October 23, 2013: About the recent notifications "outage"

Earlier this month, my hosting company (Linode) asked to migrate the main Undermine server to another rack/host in their datacenter. Whatever, no big deal. That move was completed. However, on this new hardware, problems with Linode's poor CPU speed are more apparent, and this affects my ability to send out market notifications. With 250 realms updating each hour in each region, I need to be able to check a realm for notifications in about 14 seconds on average, and after the move it's struggled to keep under 20-30 seconds per realm. The code hasn't changed, the only recent changes are that Blizzard includes realm name with every auction line (a minor but not insignificant change) and that I've been put on a new server, probably shared with more CPU-heavy nodes.

This CPU lag was more than just causing "slow" or "delayed" notifications. t's not really a speed issue like "oh, there's a new data dump, I haven't made notifications from it until 5 minutes after it was released". It's a speed issue like, "I've got 20 realms in the queue to parse, and once they're done, I've now got 25 realms in the queue." It's a little bit drinking-from-the-firehose, which means when the queue gets too long, some data dumps just don't get any notifications sent at all, so we can keep up with sending the data to the database for storage.

I don't have hard numbers on it, but my guess is that, after the migration, about 20-25% of data dumps weren't being scanned for any notifications. The system still worked, kinda, in that the next dump from a realm after a skipped one would likely get scanned, but it's not the quality of service I felt comfortable continuing to sell. That's why I disabled new (and renewed) subscriptions for a while: if I didn't have the time or an idea to fix it, I shouldn't sell a sub-standard product.

Site reach is fine. The number of unique visitors we get each day has grown until about a year ago, and it's held high and steady since then. However, the number of folks purchasing notification subs has dropped about 33% since its high a year ago. This is understandable, more casual folks playing, or more hardcore players moving on to other games, or at least people don't need notifications to make gold. I get it.

I don't have the time to mess with the site like I used to. For the first few years working on TUJ, my day job was different and let's just say I had lots of spare time. Nowadays, not so much, I don't have the time or energy to mess around with the increasingly-creaky code. So I'm not inclined to dive back in and make market notifications work even more efficiently.

So, I was thinking of dropping market notifications altogether in the next two months. Between the current issues and the slow slide of subscriptions, I figured perhaps the day to sunset notifications had come.

Since market notifications were a large part of why there's an account system, I was thinking of dropping the accounts then, too. If accounts go away, I can restore the sold stats for everyone, and just drop realm-specific addon and market API support. I'd try to keep the GE addon updated. Getting rid of a bunch of that code will not affect most users at all, and all the charts and stuff should remain. It'll take a lot of the support requirements off my plate, too.

That's how things stayed for a few days, until I had time to look at it last weekend. On Saturday, I noticed that the Realm Pop server (which is also the image server for TUJ) isn't very busy. I separated the bit that checks incoming market data against notifications from the bit that puts it into a format to send to the main auction database. Market notifications are now executed on the Realm Pop server. I watched that for a few days, noticed that SMS messages weren't sent, and fixed that as well.

Since things look more stable and consistent now, and the load is more balanced, I decided to open up for new and renewing subscriptions.

Read More »

Category: General | Posted by: admin |

August 20, 2013: Thinking about slimming down..

I doubt that this post will get noticed right away, but there are still some RSS readers diligently checking this blog, so it'll get around.

When Google shut down Google Talk in favor of Google Hangouts, it was a blow to the XMPP community, as that open standard lost its largest supporting service. The vast majority of our XMPP notifications subscribers used Google Talk, so with no other obvious choice for XMPP chat, I've decided to shut down XMPP notifications in early October. There wasn't really a monetary cost attached to running XMPP (it was free for the volume we used) but it'll be nice to have one fewer feature to support, especially since I don't personally use it.

Also in the past week I've switched TUJ and Realm Pop from using MaxCDN as a content delivery network to using no CDN at all and just hosting all static resources from Linode. I get 10x as much bandwidth from Linode per month compared to what MaxCDN gives per year anyway. All MaxCDN might give is a slightly quicker loading experience for some users far from New Jersey. I made the switch and nobody noticed, server load and bandwidth didn't notice, so after another week or so I'll shut down the MaxCDN account and save some cash there.

Market Notifications subscribers are down about 20% from their usual numbers from earlier in the year (and earlier in MoP). A drop is to be expected late in an expansion, but still, it's a bit disappointing. What can ya do.

SMS continues to be an occasional annoyance. The cell phone just sits plugged in 24/7 and occasionally it'll just lose WiFi connection or cell tower connection for no reason, and I don't notice until someone sends in a note via the contact form. SMS subcribers are also down about 30%, and between its hassle, the few people who use it, and the cost to keep it going, I'm thinking about dropping that as well. Not sure yet, but it seems likely. I'd give 2 months notice before SMS is disabled, so anyone who subscribed thinking it would be around gets their money's worth. (That's why our XMPP closure had such a long notice.)

Black Market Auction House support was a neat experiment, and it has run quite well for 6 months, especially for how hacked together and fragile it is. Unfortunately, I never found a way to support PVP realms, and Blizzard seems to have no intention of adding BMAH to the AH API. I had hoped that the BMAH notifications would bring some increase in Market Notifications subs to offset the subscription and hardware cost, but that hasn't really happened. I understand that BMAH is just a small part of the economy (and the game as a whole) but even after improving the BMAH item pages with better pricing statistics, I haven't received any feedback about it. 25% of current subscribers get the daily BMAH update, and 46% of subscribers watch for specific items to appear on the BMAH. So, all that in mind, I'm also thinking about disabling BMAH support, but that's less likely than removing SMS. Just thinking about it. It'd save me another $50/mo in WoW subs for the reporters, at least.

The Addon and the Market API don't really have any upkeep costs in time or money, despite being a bit fiddly occasionally. I had wanted to get battle pet data in both someday, but, eh. Hasn't happened yet.

What it comes down to is just cutting some unnecessary costs (CDN, SMS, BMAH subs) where people won't notice. They were parts of good experiments and learning experiences, but I do have less time to keep TUJ humming along, so I occasionally need to consider all its features and decide what's worth the time and money cost to keep going.

While I'm at it, might as well mention Realm Pop. I'm satisfied with its visit counts.. analytics shows steady growth in both uniques and pageviews, now about 2x what they were at the end of 2012. However, ad revenue remains about 20% as much as was earned at the end of that year. I guess advertisers just buy lots of ads at Christmastime and it drives the price up, or something. Regardless, Realm Pop is still profitable when compared against its server costs, and I really don't need to mess with it at all, so it'll keep chugging along.

Category: General | Posted by: admin | 1 Comment

March 08, 2013: International SMS

Our SMS Text Message service for our market notifications subscribers is now available in Canada, Mexico, Australia, Germany, Brazil, and Denmark. Normal text messaging rates from your carrier for texting the United States apply.

Unfortunately, we will not be able to offer this service to subscribers in the United Kingdom, France, Sweden, or the Netherlands.

I used T-Mobile for the text message service originally. I use T-Mobile for my personal phone. They're good folks. However, after polling our users, I've found that SMS service is a nice perk to have for the subscribers, and limiting it to the US is a bit inconvenient. T-Mobile has unlimited international text plans, but for much more money than just texting the US.. so I went with AT&T. $25/mo for unlimited US and many other countries is pretty sweet.

However, there are some gaps in coverage. The UK and France are two large gaps I wish I could text, but they're not in the list of countries included by AT&T.

TUJ sends SMS messages using a cell phone that just sits on top of the server plugged in all the time. I don't send out so many texts that I'm noticable, unless texting numbers all over the country and receiving very few replies sparks their curiosity. I'd like to use a dedicated SMS service, but, oddly enough, they're much more expensive at my relatively low volume.

edit Okay, I might as well break down the costs, since you could figure it out anyway if you wanted. And, FWIW, I never noticed an increase in new subscribers from introducing the SMS service, so it's not like this feature's making me more money than I'm spending. Basically, all it's doing is keeping current subs happy instead of bringing in more subs.

I like using prepaid phone accounts to keep costs predictable. If start getting lots of international texts (which I did, at the start, from a couple users), I'll get charged for them, since we in the US get charged for sending and receiving text messages, and I'd like not to owe the phone company hundreds of dollars. I'd just stop receiving messages that cost money when I run out of money in the account.

I used to have the $15/mo T-Mobile prepaid account with 250(?) minutes talk, unlimited US text, 100MB data. It's not on their web page anymore. Sweet deal. Technically I still have it for another week or so, the sim card is just sitting on my desk instead of inside the phone.

Now I have the $25/mo AT&T prepaid acct with 250 minutes talk, unlimited US text, unlimited text to a bunch of countries, and no data. Here's their plans. I assumed when I picked it up that it'd have the UK, but, of course, it doesn't. Whatever.

T-Mobile's international text plan is a $10/mo addon, but it requires a $50/mo minimum plan already.. so just to add international texts I'd have to pay 4x as much per month. Plus, T-Mo isn't known for being on the ball with much of anything (coverage, speed, etc isn't much to write home about) so I didn't expect their international text plan to be any better, but it kind-of is. They have 200+ countries (instead of AT&T's 100) including Canada, Mexico, Australia, Germany, Brazil, United Kingdom, France, Sweden, Denmark, and the Netherlands.

It's hard to judge what to do. I'll throw in another $10/mo just to support Canada and Australia. But do I wanna throw in another $35/mo for the UK and France? If I added them, would I get (and keep) another 7 subs from those countries to cover the cost? Looking at my numbers, probably not. It's too bad, I'd like to do it, but it's not worth the money.

Category: General | Posted by: admin |

February 16, 2013: Black Market Update

We have chosen not to pursue reporting on the Black Market Auction House for PvP realms. We cannot reliably report on those realms when our characters are getting killed. If you don't like it, write to your congressman Blizzard forums and ask that the Black Market Auction House be included in the Community Platform API. Same goes for all EU realms.

Subscribers to our Market Notifications service can view past auctions on our Black Market Auction House page, found under the Great Deals section.

Items which are found on the BMAH now have their own item page, and can be searched just like any other item. If an item can be found on the BMAH, subscribers to our Market Notifications service can view current and past BMAH auctions of that item across all the realms we monitor.

If you have any market notification set on an item, you will be notified if that item appears on the BMAH. The criteria of that notification (quantity available, price to buy X, etc) don't matter for the BMAH messages. You can watch just your own realm, or make it "global" to watch all realms that we monitor (US PvE and US RPPvE), just like we can for all other items.

If you have any questions or suggestions regarding the Black Market Auction House support and features, as always we recommend you visit our forum. Thank you.

Category: General | Posted by: admin |

February 11, 2013: User Poll - SMS Text Messages

For the past few months, we've explored sending SMS text messages to subscribers of our Notification Events system. This opt-in communication method would sit alongside our existing EMail, RSS, and XMPP notification methods. Users would, within a minute, receive on their cell phones the subject line of notifications as they were created, along with a short link directly to the text of each message.

Our beta testing of SMS Text Messages has gone well so far. Because of the high cost of sending international texts with our current provider, we've been limited to serving US cell phones. However, we may be able to support international phones soon, depending on feedback from this poll:

How useful are SMS Text Messages from The Undermine Journal to you? This poll is open to every user, so please respond regardless if you are a current SMS Text Message user, and regardless if you are in the United States. We want to hear from everyone. Please log in to cast your vote.

Category: General | Posted by: admin |

January 26, 2013: Black Market Auction House

The Undermine Journal has started dedicating resources to embed reporters in the The Veiled Stair of Pandaria to keep you informed about what's available at the Black Market Auction House.

These reporters do not use the same methods as our sources in the Alliance, Horde, and Neutral auction houses, so, due to their remote location, our communications with them are intermittent. We strive for hourly updates, which is the same frequency we get with other auction houses.

To get this data, we need a character to log in to World of Warcraft and talk to Madam Goya once each hour.

We have all US PvE (RP and Normal) realms covered. We're trying a few PvP realms, but due to the nature of PvP, our characters that survey the auction house are getting killed. So, if you see a level 1 female Dwarf suddenly appear, don't be alarmed, she doesn't have any gold and doesn't want any trouble.

You can see the results of our work on our front page, as well as the "Black Market Auction House" link under the "Great Deals" heading.

Category: General | Posted by: admin | 3 Comments

January 03, 2013: Fun times.

Looking thru my referer list, N3llee's twitch.tv channel is up on the charts. I decide to check it out, and the channel was streaming live, and was looking at TUJ at the time. I decided to see if it was "really live", so I changed the site to send an error page: "Hey N3llee, just wanted to see if this was live, thanks."

Host freaks out a bit, stunned, tries to find out who's responsible. I can't type in the twitch.tv chat b/c I don't have an account and don't want to be bothered. So I change the message: "lol, it was me, just messin' with ya --Erorus" and it was kinda comedy gold.

Here's a link to the vid right before I start.

See below for the chatlog (which doesn't include the host's on-air comments)...

Read More »

Category: General | Posted by: admin |

January 01, 2013: Map of SMS Users

I like to see how far and wide my work has spread, but it's difficult to do cheap mass geolocation by IP that's more specific than country.. so here's the next best thing: a map of SMS users, approximate location based on area code.

Category: General | Posted by: admin |

November 20, 2012: Child's Play



Most of you are probably already familiar with the Child's Play charity and the generous work they do. As Thanksgiving (and the subsequent shopping frenzy of Black Friday) approaches, we encourage you to take a moment to remember other gamers in need, and to make your donation to Child's Play.

For our part, The Undermine Journal has donated $250 to the charity this year, funded by your interest in our Notifications Events services. Thank you for your support which allows us to make such a generous donation.

Category: General | Posted by: admin |

October 20, 2012: Ads

Earlier this month, I asked for feedback about switching to an ad-supported model. You can read how that discussion went in the forum. Financially the site is stable, no problem. Subscriptions cover the cost of development and upkeep, and they provide some income to try new things (like SMS notifications). I don't like ads, but lots of folks like market notifications, and I'd like to remove that paywall if I could. One way or another the site needs financial support, so it's either we keep the subscription setup or we use some ads.

The plan I ended up wanting to try was to keep paid subscriptions for seller notifications, rare item notifications, RSS, XMPP and SMS delivery methods, but to make regular item notifications via email free of charge. Logged in paid subscribers would never see ads on the website.

I planned on using Google AdSense, like I have on Realm Pop. However, AdSense doesn't officially support HTTPS (SSL) websites, and The Undermine Journal is exclusively served with SSL. (There are a bunch of reasons for this, none very important, but the biggest reason I want to keep it is to help protect people who use the site from work.)

So, AdSense was out. I went looking for ad providers that support SSL. BuySellAds looked interesting. Instead of the adsense model, where Google crawls your pages and then just picks ads based on that, BSA lets you work more directly with people buying ad slots. You show buyers your website and the slots, put a price on it (either by impressions or a flat monthly fee or something), and buyers can buy ad slots and BSA will stick their ads in there. I liked the idea, it gave you more flexibility with pricing and you can directly approve/deny ads that run on your site.

I sent BSA an email describing my site. Someone replied back and said my stats and content should be able to support being a BSA ad slot seller. Then I filled out the online form to be an official seller, and a few days later they responded that they did not approve my website. So I couldn't sell ad space through them after all. Boo.

I took another look at AdSense. The problem with SSL is that they want to serve images from a non-SSL server, which causes your web browser to freak out. However, someone discovered that text-only ads work fine, so that's what I enabled on the front page.

The front page doesn't get most of our traffic. Item pages by far get the most hits. Unfortunately, those generated pages make for poor AdSense pages. The text on them doesn't really change (just numbers and lists of items), they aren't very descriptive, and Google would need to crawl all of them, which I currently deny since it would put extreme load on my servers. So, all I can do is put text ads on the front page.

I've placed ads on the front page where the realm forum listing would go. It only shows for non-logged-in users. Nothing else has changed yet; I want to see what people really think of seeing ads and what kind of return I get for their placement there. I don't really like seeing them there, and I block `em. You can block ads too, I don't care. :) I'm concerned I'll get lots of gold selling ads and ads that aren't relevant at all and it'll look ridiculous. We'll see how it goes.

Category: Money | Posted by: admin |