Barracuda ADC Load Balancer – How to show client IPs and not the proxy IP address – Part 1

In order to see the client origination IP address on the real back-end servers when a Barracuda ADC load balancer is used there are two options.  The first is to use layer 4 load balancing on the service type.  The other option is to use a layer 7 proxy service and enable client impersonation (while setting the default gateway to the ADC - but this causes other problems that I’ve discussed in part 2 of this post).  Both of these options work ok for clients outside of the back-end server network, but these do not work if the clients are in the same layer 2 network.  (For example if your virtual IP is 10.0.0.5 and the back-end servers are 10.0.0.10, 10.0.0.11 and your client is 10.0.0.100, then neither of these options will work).  This is because when the back-end server sees that the IP address is on the same network, it will not send the return packets back through the Barracuda ADC, but rather straight to the client and thus break the TCP stream. The solution to this is to use a layer 4 TCP service with “Direct Server Return” (DSR) enabled, plus a loopback interface must be added to each of the back-end servers that have the IP address of the service virtual IP (VIP) assigned to them.   DSR will cause the ADC to change the data frame’s MAC address to the MAC of the real server before placing the packet back on the wire.  The server will accept the packet (bound for the service VIP) because the loopback has been assigned the same IP.  The server will then respond back directly to the client using the VIP as the source address. To do this, first go to the device manager and right click...

Speed up mailbox migration to Exchange 2013 – They’re too slow!

Moving mailboxes from Exchange 2007 or 2010 to Exchange 2013 can often go very slowly, even when the network and server resources are fast and abundant!  The Exchange Mailbox Replication Service (MRS) has extensive resource throttling enabled by default in order to prevent mailbox moves from choking out the rest of the users.  Because of this you may see mailboxes with a status of RelinquishedWlmStall and if you look at the details of the Get-MoveRequestStatistics report you will see mailboxes have a lot of time sitting idle under the TotalStalledDueToWriteThrottle counter. Microsoft tech support suggests making changes to the “MSExchangeMailboxReplication.exe.config” file located at “C:\Program Files\Microsoft\Exchange Server\V15\Bin”.  The values to look at, along with their default settings are: MaxActiveMovesPerSourceMDB=”20″ MaxActiveMovesPerTargetMDB=”20″ MaxActiveMovesPerSourceServer=”100″ MaxActiveMovesPerTargetServer=”100″ MaxTotalRequestsPerMRS=”100″ ExportBufferSizeKB=”512″ We typically like to set these values so that about 10 mailboxes can be moved simultaneously.  The ExportBufferSizeKB we’ve used in the past is “10240”.  The Exchange Mailbox Replication Service should be restarted after these changes. The other suggestion Microsoft has made is to disable content indexing on the target database so that the search index scanner isn’t overwhelmed by all the new messages needing to be indexed.  You’ll want to set it back once the migration is complete. Set-MailboxDatabase “DB1” -IndexEnabled:$False In our experience however, these first two suggestions do NOT have tremendous impact on the overall speed.  The following two options have proven to be the most effective for us. Use the “-priority emergency” parameter on the mailbox moves.  This will give the move the highest priority in the MRS queue.  For example: New-MoveRequest -Identity “user@domain.com” -TargetDatabase “DB1” -Priority emergency If the priority flag and the MRS config editing doesn’t make the moves fast enough for you, then disable MRS throttling altogether!  To...

NetFlow setup for 3750x with C3KX-SM-10G

The Cisco 3750x switch does not support NetFlow natively, but the C3KX-SM-10G module has ASICs that support NetFlow.  With the C3KX-SM-10G module, NetFlow can only be run on the 4 interfaces in the module — this does not add NetFlow on the entire switch. It is however possible to capture the switch’s traffic and port mirror (SPAN) it over to one of the C3KX-SM-10G interfaces so that it can be exported for NetFlow.  The problem is that in order to mirror traffic to a port and export NetFlow, the port must be in an UP state.  In order to force the port up we took an LC fiber patch cable and split apart the plastic end and pulled the cable apart so that we had 2 single fiber strands.  Then we plugged in an SFP and connected the port into itself by looping the single fiber back to the same SFP.  Use caution with this as it will create a loop — it might be better to setup the mirror first as it will put the port in an UP/DOWN state that I’ll mention later. First setup the port mirroring selecting the source VLANs or interfaces.  Also point the SPAN at the interface where the fiber loop is on the C3KX-SM-10G. monitor session 5 source vlan 1 - 5 , 7 , 100 monitor session 5 destination interface Gi4/1/2 Then setup the NetFlow export.  Start by defining the flow monitoring records. flow record NETFLOW  match datalink source-vlan-id  match datalink dot1q priority  match datalink mac source-address  match datalink mac destination-address  match ipv4 version  match ipv4 tos  match ipv4 ttl  match ipv4 protocol  match ipv4 source address  match ipv4 destination address  match transport source-port  match...

Barracuda ADC Load Balancer with OWA – Login loop

We had a problem with Barracuda’s ADC load balancer and Outlook Web App for Exchange 2013.  The user would get to the OWA login screen and login only to be brought back to the login screen again in an endless loop.  We were using the “Instant SSL” service type as described in the following Barracuda document: https://techlib.barracuda.com/adc/msx2013deploy The problem was that “Rewrite Support” was enabled on the service.  Disabling this allowed the users to login without...

Exchange 2013 DAG: Can’t add a database copy – “Seeding operation failed”

We had an issue when attempting to add a database copy for any mailbox databases in a new DAG.  We received one of the following errors every time, regardless of the source or destination server. The seeding operation failed. Error: An error occurred while running prerequisite checks. Error: The specified database isn’t configured for replication and therefore cannot be used to perform seed operations. The seeding operation failed. Error: An error occurred while performing the seed operation. Error: An error occurred while processing a request on server ‘MailboxServerName’. Error: Database ‘6060c9ac-363a-4e52-a02e-ba749625e8ea’ was not active on source server ‘MailboxServerName’. [Database: Test3, Server: MailboxServerName.domain.com] After closing out the dialog box we found that the database showed multiple copies, but that the copy was in a failed and/or suspended state. We troubleshot this and found that the client’s several domain controllers were not replicating information as quickly as we were used to.  To help with this, we set the preferred domain controller to a global catalog DC residing in the same network using the following command:  Set-ADServerSettings -PreferredServer server.domain.com This seemed to help some other minor ECP errors where we felt like IIS was thinking faster than AD could keep up, but this did not remove the errors we saw when creating a database copy. The solution was to walk away and wait for about 30 minutes and on return we found that some of the mailboxes cleaned up on their own and went to a healthy state.  For the majority that didn’t resolve themselves we fixed them in ECP by selecting the database and on the right pane click “Update” under the database copy that failed. In the dialog box that followed we did not specify a source server, continued, and the mailbox replicated...

Exchange 2013 – Can’t access ECP, 500 Unexpected Error

We had an issue with a client migrating from Exchange 2007 to 2013 where we couldn’t access ECP on any of the new 2013 CAS servers.  Instead we received the following: 500 Unexpected error  : (  An error occurred and your request couldn’t be completed. Please try again. We also noticed that although the Exchange Management Shell would successfully open and operate, on first opening it we received an error that contained the following message: Unable to determine the installed file version from the registry key ‘HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\PowerShell\1\PowerShellEngine’.  After much troubleshooting (reinstalling CAS servers, deleting/recreating ECP folders, checking permissions, etc.) we found that we needed to install the “Windows PowerShell 2.0 Engine” feature from Server Manager (this also installed .NET 3.5).  These features needed to be installed on ALL of the new 2013 Exchange servers and not just the CAS servers. Although these features aren’t a prerequisite for a 2007 to 2013 migration, there must’ve been something in the existing AD/Exchange environment that required it.  Hope this helps someone...

Ubiquiti AP powers up & reachable via wifi, but no link on LAN

We’ve seen this happen twice recently with different customers so thought we should post our solution.  We have seen where a Ubiquiti AP (NanoBeam & NanoStation) will power up via the power injector and the unit is reachable via wireless, but there is no link light on the switch or to a laptop plugged directly in.  In both cases the units were mounted up on a roof, so we first replaced the cables from the injector to the switch but nothing changed.  We found we had to either re-terminate the RJ45 ends on the cable going from the injector up to the roof or replace the cable completely before it started working again.  The cable end or cable itself was damaged so that it wouldn’t allow link, but PoE passed through....

Adding PoE to a Cisco 890 Series Router – (C891FW-A-K9)

In order to add PoE to an 891 router, the datasheet says you need “800-IL-PM-4 with 125W PSU.”  I had to use the Cisco Commerce Workspace to generate a config starting with the 800-IL-PM-4= part number and then added the 125W power supply there.  The following is what was shipped: The 800-IL-PM-4= part is a daughter-card that is installed onto the motherboard of the 891 router. The 125W power supply had the Cisco part number of PWR-125W-AC.  The power supply is a LiteOn unit, model PA-2121-1-LF or 341-0502-01 (both were listed) putting out 12V, 3.5A of DC power on one rail and 53.5V, 1.55A on the other. Lastly, there were three baggies with hardware. 50-1807-02-A0 - 2 standoffs with two notches 50-1700-01-A0 - standoff with one notch 48-0421-01-A0 - 3 screws  ...

Outbound faxes from a Dialogic SR140 not using T.38

Had an issue with an IP fax server (RelayFax with a Dialogic SR140 connecting via SIP trunk to an Adtran SBC) where inbound faxes were working fine and using T38 but outbound faxes were never negotiating T.38 and falling back to G.711 passthrough.  As a result we were having some unreliable outbound faxes. In the Brooktrout Configuration Tool under the T.38 Parameters tab we had to change the delay time for media renegotiation.  The “Media Renegotiate Delay Outbound, msec” was set to “-1” and I changed this to 5000 msecs.  See...

Remove AddRules.htm from inbound faxes for RelayFax

By default RelayFax will add an attachment to inbound faxes named AddRules.htm.  This attachment is a page that will allow users to mark certain fax senders as spammers.  In cases where these emails will be picked up by copiers/printers via email, then this attachment can be problematic.  In order to remove this attachment do the following. Stop the RelayFax service Go to “C:\Program Files (x86)\RelayFax\App” and edit “NewFax.dat” and “PartFax.dat” You will find a line with “$INCLUDEURLS$”  - Put a pound sign (#) in front of that line to comment the line out and save the files Restart the RelayFax service...