Update for velvet.net users.

[reddragdiva]

After twenty-eight hours of most interesting efforts to get stuff shifted from Websynthesis to Wefixtech, we have everything in place on Wefixtech and just need to shift the MX record over ... This involves getting Demon to shift it over, which is not feasible until mid-January. (Arse.)

The problem is that topbit has been having problems with mail on Websynthesis. Yesterday, we were told that Velvet needed to go, now. (Which was rather unfortunate timing.) This evening, he says that he has put into place valid user check (so that opportunist spam gets rejected immediately), and that the mail situation is now much better. What he says he won't guarantee as of tonight is that he won't feel he has to do this again.

Since we can't get any assurance that the mail will get through, at the moment we're looking at narnee having to go back to Glasgow immediately to sort this out. Which trashes her holiday and pretty much everything she wanted to do with it. This is, understandably, less than ideal. So we still hope to sort out things at least keeping working until mid-January, when it can be shifted from Websynthesis to somewhere more capable, so that this doesn't have to happen for your stuff to keep working.

Our apologies to Velvet users*, we're doing our best and will keep you updated!

of which there are probably more on reddragdiva than any other journal

Comments

[topbit.livejournal.com]

it's not spam per-se, mostly false-From: bounces. They are sent to a user that doesn't exist on velvet, so the system was bouncing them back, and mostly getting a double-bounce as the destination didn't exist in the first place.
So every email was generating at least twice as much bandwidth and IO/slots plus landing in the postmaster account - the end result being that the other 2% of valid mail was getting crowded out and couldn't even get into the system in a timely manner - sometimes with an email taking five hours or more to finally squeeze in.

with the mails now not even getting there (which was sometimes thousands per hour), there's now room for the good stuff to get through quickly - as it should.

If that continues to be the case, then there shouldn't be a problem. In the meantime, I'm going to check it regularly, and make sure it's OK, and do what I can to work around the issues as required. Crunchtime comes when everyone is back at work, as thats when people want their emails quickly.

[reddragdiva]

If I can get onto hawkeviper about some way to just forward the stuff to Wefixtech so that it's accessible - just passing into Websynthesis and out again shouldn't stress it appreciably - then that should solve things until Jan 15th.

If you can just contact us with the issues as they arise so that we can work through them, such that the plug does not have to be pulled before about Jan 15th, that will be sufficient for us to work with! Tony knows his system inside out and iio.wefixtech.co.uk is a shiny new dual Opteron with all the salad on a massive connection, so it should handle all this just fine.

I'm leaving for Whitby tomorrow, which sucks arse in this situation, but I am in mobile contact (number in memories, friends accessible) all that time ... not sure about email, depends if there's wifi I can leech off ;-)

But there's enough sysadmins around this problem we should be able to keep the mail moving and the velvet users in touch with the world without massive trauma. Which is the point, of course :-)

[topbit.livejournal.com]

I can't see a difference between having it land on websynthesis and landing in a pop3 account or getting forwarded on somewhere else. If anything instantly forwarding it may be more work.

For now, I'd leave it where it is, at least till I know for sure what the situation with the mail traffic is, good or bad.

[reddragdiva]

And I shall send out emails now so we have admins on this side who can make sure The Mail Gets Through!

[dan-lane.livejournal.com]

This is an unfortunate side-effect of running a mailserver on the internet in this day and age.

From the sounds of it you're performing user validation after accepting the mail... to avoid this (and hopefully bring the load down on the machine) validate a user before accepting the mail and return a 550 error if the user is not known. This will also avoid nasty secondary bounces.

This method limits some of the more flexible things you can do but is worth looking into if you have a machine that is buckling under the load caused by false spam headers.

Here comes the science bit

[topbit.livejournal.com]

That's what I've started to doing now - validating (at least some) inbound addresses early in the connection (after it's passed an RBL check). I've swapped the original qmail-smtpd with magic-smtpd and got a little shell-script/grep checking for valid velvet.net users (then it double-checks any fails via the vpopuserinfo). If it's not on the list, it's not coming in.

My general mail logs are rotating too quickly to give a long-term stats (I've just increased 4x the amount kept), but from 9am till 11:30 the system has blocked 1,160 connections. That is probably a little more than the sbl-xbl.spamhaus.org has helped to block in the same period, but those 1,160 were all from valid MTA, just not to valid users.

Well, that's definately another piece of software to be installed on a new system!

Re: Here comes the science bit

[stevenothing.livejournal.com]

We had that problem with qmail. Massive problem, in fact. Replaced it with the magic smtpd as you did.

If you're using vpopmail, I've got a little C program I wrote which links against the vpopmail libraries that you can use as your user validation script. I found it to be about a hundred times faster than the shell script I was using, especially after the shell script was filled up with various nasty hacks to make ezmlm and the various levels of .qmail-fish-default files work.

Disclaimer: I can't verify that this is 100% bug free or anything. But it's been running succesfully for over a year on a machine with 1700+ domains on it, and has so far proved itself to be better than any other way I've found of getting qmail to refuse email for invalid users at SMTP time.

Long term, (and preferable) solution, I use postfix :)