Applications are stronger when they exercise.
This post is comes in response to a blog post that was sparked from a twitter post. The blog post “My wish for 2015 - better tooling to provide better insight” by Frank Denneman. The tweet, from what I can see, started by @josh_odgers included the meme below.
I will not regurgitate what was said by Frank because I agree with his points of view. In general, you should know as much about your infrastructure as possible but it can’t end there. Infrastructure design and configurations are rarely the same. Having the tooling that gives greater insights into your infrastructure is only part of what’s needed when new or updated applications are deployed. While I to wish for better infrastructure insight, I also wish for ways to exercise applications.
Another reason to not have the application developer or service owner dictating infrastructure details is because more gains can be had by tuning the application or service. This is where they should focus their time! I’m not a programmer but I think many would agree that the user experience is key. Applications can either “fly or die” depending on the application experience from the users point of view.
One thing I see missing from most deployments processes is a way to exercise the application or service before it is released into production. If you can’t generate or simulate load how can you, with any amount of accuracy, provide infrastructure requirements. Functional testing is necessary but it’s useless to the infrastructure team. Too many times have I seen applications deployed into production only to perform poorly or cause downtime. Or on the flip side, too much infrastructure was purchased to support the application or service. Both scenarios are bad.
I understand that this request does not come easy. I also understand that this won’t fit all application deployments because the cost, time and resources, may be to great. My question though is why? I would think having this would strike the right balance between spending too much on infrastructure and time wasted troubleshooting performance issues.
My hope in 2015 is that we find better ways of exercising our applications and services preproduction while capturing better insights at the application and infrastructure layers so that we deliver stronger end to end solutions to our customers.
4 Comments
Add yoursComments are closed.
Nice post!
Concur that the app owners should be informing the data and information infrastructure people about application service requirements in terms of Performance Availability space Capacity and associated security or other considerations. For example the app owners should have an idea of Performance in terms of transactions (e.g. data rates which can translate to IOPs) and transaction size (IO size), types (reads queries, rand or long sequential table scans, updates, loads, inserts, etc). App provider should also provide service level objective (SLO) indicators for performance response times along with growth projections.
Then there is space needs for primary database tables, index, journals, redo or other logs, as well as scratch / ephemeral space for import, exports, work area along with other needs.
In addition the app owner should provide Availability requirements including security, access control, compliance needs or concerns, frequency, coverage, retention, frequency of protection, RTO and RPO for the entire application from a business point of view, as well as for different components (e.g. app, database, servers, storage, network, etc).
And of course, instead of the app owner saying this much flash SSD or that much RAID 0/1/10/4/5/6/FEC/Erasure code or that much block, file or object storage, they should communicate the above needs in service, app and business context to let the data and information infrastructure people make informed decisions. However there is a catch, the catch is that the data and information infrastructure people need to start speaking the language of the business, applications and service requirements.
This means asking the right questions or facilitating discussions, educating consumers of their services with types of info needed. As long as the data and information infrastructure people speak IOPs, RAID levels, block, file, object etc… the app people will make requests, suggestions or dictate in those terms.
Take some tips from and learn what cloud service providers do, how they offer or promote service offerings (besides cost) to decision makers.
Btw, in addition to not having app owners dictate technology selection requirements, by getting to a neutral or abstracted discussion around business and app service requirements, you might also clarify needs vs. wants. For example, the app owner wants RTO and RPO of 0 and SSD fast response time, however during discussions you learn what they and the business needs are RTO of hour or more, RPO of 0 for some things, RPO of a few minutes for others among other revelations.
I love the idea of wide spread load testing of applications, but part of the problem that I have seen is load testing is a complex and expensive endeavor. One that is for some reason or other never budgeted for in either time nor cost of an application deployment or creation. If we could make the load testing process easier or bring about an intelligent solution that could look at the app it was testing and determine itself a sequence of tests to run and then easily scaleramp up it’s testing that would be amazing. Products like Load Runner need specialists to help implement and design the testing scenarios and tons of resources to properly run them.
I have worked for many global corporations supporting migrations from physical to virtual or v2v and the majority of this corporations had trending information collected over the years on basic metrics like CPU, memory, disk and network usage. When analyzing this information it was relatively easy to size VM’s correctly and deployment was succesfull. Point being: application performance benchmarking is out there and being done - it’s not uncommon. Application load testing is necessary for apps for which you predict a pattern of growth that is an unknown, this is not the case for most legacy apps. Therefor I would make the case that application load testing is crucial for greenfield deployments, less crucial for existing apps with a known growth pattern.
Good point Benjamin but it’s not just about sizing the application. Load testing also put’s load on the infrastructure. And if done programmatically frees up resources (x number of people) from having to manually log into the application to generate load. This could bring bottlenecks to the surface that would’ve only been seen in production. In the migration scenario you could get the performance profile of the application but what you miss is it’s direct impact to resources “end to end” in the new environment.
In the case where you can capture the performance profile I think it would be nice to be able to replay that performance profile, anywhere I like, which would also generate or simulate the load put on the infrastructure without having to create and install applications. Maybe I’m thinking outside the box here but it seems within the realm of possibility.