Yes, I actually do. It’s a behavioral driven testing framework. I love the idea of it but hate working with it because everyone uses it improperly.
What the steps in a scenario should look like:
Given a user has a bank accout
And the account has a balance of $10
When the user attempts to withdraw $20
Then the transaction should fail
How everyone I’ve worked with uses it:
Given the system is setup
Given the user TESTUSER1
Given load TESTDATA1
When the user sends a request
ACTION | AMOUNT
WITHDRAW | 20.0
The the response should be 400
STATUS | MESSAGE
Failure | Not enough funds
On that same page is Gherkin which is what both examples are. (I may have gotten the syntax slightly wrong.) Cucumber uses Gherkin. I forget which is which exactly. Maybe Cucumber is the code that reads Gherkin and executes step definitions.
For whatever reason, people try to make a small number of extremely flexible step definitions which leads to scenarios that don’t actually read as business requirements. (Which is the entire point in the first place.)
Given a user has a bank accout
And the account has a balance of $10
When the user attempts to withdraw $20
Then the transaction should fail
And the account should have a balance of $20
Given a user has a bank accout
And the account has a balance of $10
When the user attempts to withdraw $5
Then the transaction should succeed
And the account should have a balance of $5
Given a user has a bank accout
And the account has a balance of $10
When the user attempts to deposit $5
Then the transaction should succeed
And the account should have a balance of $15
Doing something like this is more elegant. The steps definitions would be
a user has a bank account performs setup tasks to load a test user into the system.
the account has a balance of X would either load the account with that much or just make an initial deposit.
the user attempts to withdraw X/the user attempts to deposit X would both perform those actions. If it’s a web API they’d be HTTP requests.
then the transaction should X would check HTTP status code. You could make this two separate step definitions or have logic based on the word. Two separate step definitions is probably better.
the account should have a balance of X checks the balance in the account.
Sure, and my issue isn’t that it’s capable of my requirements, but simple to use and meets those requirements. Simplicity is a quality all its own.
JUnit provides sufficient output to meet my requirements, so why overcomplicate it w/ Cucumber? AFAIK, cucumber tests in Java output JUinit as well (at least ours does), so what value exactly is Cucumber providing?
If you don’t want the scenarios that are readable in a business requirement format then it’s not worth the trouble. I haven’t ever seen anyone use it in that way.
Yeah, I don’t get it. The only people looking at the results are devs, so we really don’t need the business requirement text.
I could maybe understand if product owners were creating the requirements in the feature files and devs were implementing them in the code, but that never happens. So the whole cucumber library thing we have going on just feels annoying.
It’s not the only thing we over-engineer. We have dependency injection when we only ever have one dependency (except in tests, in which case we could just use mocking). We take microservices to the extreme (we just broke out a couple hundred lines of code from a few thosand lines of code service). Our test code is unnecessarily reusable (test code is one place where DRY doesn’t really need to apply). And so on. This one in particular is especially annoying because it makes it harder to find the code without providing much benefit.
Our QAs seem to like it, so whatever. I’ll still complain though.
These are for a mix of end to end and integration tests.
I mostly do unit tests as a dev, so our tests are simple enough that they don’t benefit from more structure than being grouped by suite. E.g.:
AuthService
valid user creds can login
invalid user creds cannot login
non-existent user gets same error as wrong creds
UserSettingsService
can change language
cannot set empty password
These don’t have long flows, so there’s no benefit to documenting steps (they usually have one step).
My complaint about cucumber/gherkin isn’t with documenting steps, it’s with managing them in separate files. We have a Service.feature file that documents the scenario and the ServiceTest.java that documents the steps. I don’t see the point in having those be separate files, especially since the only people defining inputs and scenarios are the devs (dedicated QA in our case). We occasionally have our BE devs help write a few tests, and every time it’s a struggle for them to figure out where everything is. It just feels over-engineered.
In unit tests, we parameterize our tests just like with cucumber, we just do so in the code. E.g. in Python:
@parameterized.expand([(1, 2), (2, 4)])
def test_duplicate(num, exp):
res = dup(num)
assert res == exp
I would much prefer something like that in our end to end and integration tests.
Yes, I actually do. It’s a behavioral driven testing framework. I love the idea of it but hate working with it because everyone uses it improperly.
What the steps in a scenario should look like:
How everyone I’ve worked with uses it:
Your (verbose and harder to understand) example reminds me of https://jbehave.org/reference/stable/story-syntax.html
On that same page is Gherkin which is what both examples are. (I may have gotten the syntax slightly wrong.) Cucumber uses Gherkin. I forget which is which exactly. Maybe Cucumber is the code that reads Gherkin and executes step definitions.
For whatever reason, people try to make a small number of extremely flexible step definitions which leads to scenarios that don’t actually read as business requirements. (Which is the entire point in the first place.)
Doing something like this is more elegant. The steps definitions would be
a user has a bank accountperforms setup tasks to load a test user into the system.the account has a balance of Xwould either load the account with that much or just make an initial deposit.the user attempts to withdraw X/the user attempts to deposit Xwould both perform those actions. If it’s a web API they’d be HTTP requests.then the transaction should Xwould check HTTP status code. You could make this two separate step definitions or have logic based on the word. Two separate step definitions is probably better.the account should have a balance of Xchecks the balance in the account.Maybe that’s why I hate using it, or maybe I just hate testing frameworks that do more than run tests. Here’s what I want from a testing framework:
Our QA uses cucumber and it works for them, so I only whine when I need to deal with it.
Cucumber can do all of that. Except for failures over time but I’ve never used a framework for that. CI tools typically track that.
Sure, and my issue isn’t that it’s capable of my requirements, but simple to use and meets those requirements. Simplicity is a quality all its own.
JUnit provides sufficient output to meet my requirements, so why overcomplicate it w/ Cucumber? AFAIK, cucumber tests in Java output JUinit as well (at least ours does), so what value exactly is Cucumber providing?
If you don’t want the scenarios that are readable in a business requirement format then it’s not worth the trouble. I haven’t ever seen anyone use it in that way.
Yeah, I don’t get it. The only people looking at the results are devs, so we really don’t need the business requirement text.
I could maybe understand if product owners were creating the requirements in the feature files and devs were implementing them in the code, but that never happens. So the whole cucumber library thing we have going on just feels annoying.
It’s not the only thing we over-engineer. We have dependency injection when we only ever have one dependency (except in tests, in which case we could just use mocking). We take microservices to the extreme (we just broke out a couple hundred lines of code from a few thosand lines of code service). Our test code is unnecessarily reusable (test code is one place where DRY doesn’t really need to apply). And so on. This one in particular is especially annoying because it makes it harder to find the code without providing much benefit.
Our QAs seem to like it, so whatever. I’ll still complain though.
I think there’s one question you should answer in order to fully describe your “testing framework”: is it being used for “end to end tests” or “integration tests” or “unit tests”? https://k-hartanto.medium.com/testing-pyramid-and-testing-ice-cream-cone-what-is-the-difference-6ddde3876c20
For unit tests, something like https://en.wikipedia.org/wiki/JUnit is useful. For testing a program after it’s deployed, something like https://jbehave.org/reference/stable/story-syntax.html is useful. You get different information about your program from each type of testing, and one type can detect issues even if the other didn’t, so doing both is useful.
These are for a mix of end to end and integration tests.
I mostly do unit tests as a dev, so our tests are simple enough that they don’t benefit from more structure than being grouped by suite. E.g.:
These don’t have long flows, so there’s no benefit to documenting steps (they usually have one step).
My complaint about cucumber/gherkin isn’t with documenting steps, it’s with managing them in separate files. We have a
Service.featurefile that documents the scenario and theServiceTest.javathat documents the steps. I don’t see the point in having those be separate files, especially since the only people defining inputs and scenarios are the devs (dedicated QA in our case). We occasionally have our BE devs help write a few tests, and every time it’s a struggle for them to figure out where everything is. It just feels over-engineered.In unit tests, we parameterize our tests just like with cucumber, we just do so in the code. E.g. in Python:
I would much prefer something like that in our end to end and integration tests.