Improve feedback loop for AI agents with custom linter

AI agents are really great — there's no doubt about it. That said, one thing I'm repeatedly seeing is that their output is highly dependent on the existing codebase and the quality of the feedback loop. Files like CLAUDE.md, AGENTS.md, skills, and memories are important, but the feedback loop matters more than any static instruction. Just to give you an example. At Ren, at some point, we had 3 instructions telling Claude not to use inline imports:

6. Add imports on top of the module. Never add them as inline imports.

...

**IMPORTANT: Don't use inline imports. Imports should always be at the top of the module.**

...

**DO NOT USE INLINE IMPORTS!!!**

Guess what? Claude Code was still often using inline imports. So, how to improve the feedback loop to avoid that? How to catch such violations without manual review?

Automate quality checks

The best way to give an AI agent feedback is to provide it with an executable tool it can run after it's finished making the changes. For example, instead of hunting down inline imports during code review, build a tool that the AI agent can execute. A tool that will tell the agent "You should not use inline imports!" and exit with code 1.

A great start is to add code formatting, linting, and security checks (e.g., to detect hardcoded passwords) to your codebase. They should run as part of every CI/CD pipeline - PRs with failing checks should not be merged. Nowadays, ruff covers most of these - linting, formatting, and a subset of security rules via the S ruleset. So if you don't know where to start, simply install ruff and run it as part of your CI/CD pipelines.

You can read more about code quality tools here

Take a step further with a custom linter

Adding ruff is great, but it's not enough to move fast in the AI agent era. Tools like ruff check for common standards defined inside PEP-8 and broader - a great first step, but only a first step. My idea is simple: encode as many of your coding standards as possible into custom linting rules. Want to avoid inline imports? Implement a rule. Want to avoid IFs inside tests? Same! Want to enforce the usage of your company's domain for email addresses in tests? Same! Want to prevent variables named jan? (Yes, that's me.) Go ahead, add a rule!

With that in mind, I built jg-lint - "Extensible Python linter with a Rust core." It provides all the boilerplate code for a linter, along with some built-in rules, while allowing you to define your own rules.

You can install it like any other Python package: uv add jg-lint (requires Python >=3.12).

Implement your first custom rule with an AST parser

Let's say we really hate inline imports. So let's implement a rule for that.

First, add jg-lint to your project:

$ uv add jg-lint

Second, configure jg-lint inside pyproject.toml:

[tool.jg-lint]
rules_path = "./rules"

This way we tell jg-lint to find rule definitions inside the rules directory.

Third, we need to create the rules package (folder with __init__.py) and add a no_inline_imports.py module:

import ast

from jg_linter import Rule, Violation


class NoInlineImports(Rule):
    code = "JGL001"
    message = "Inline import; move it to the module top level"

    def check(self, file_path: str, content: str) -> list[Violation]:
        if not file_path.endswith(".py"):
            return []
        try:
            tree = ast.parse(content)
        except SyntaxError:
            return []

        violations: list[Violation] = []

        def visit(node: ast.AST, inside_function: bool) -> None:
            if inside_function and isinstance(node, (ast.Import, ast.ImportFrom)):
                violations.append(
                    Violation(
                        file_path,
                        node.lineno,
                        node.col_offset + 1,
                        self.code,
                        self.message,
                    )
                )
            in_func = inside_function or isinstance(
                node, (ast.FunctionDef, ast.AsyncFunctionDef)
            )
            for child in ast.iter_child_nodes(node):
                visit(child, in_func)

        visit(tree, False)
        return violations

So what's happening here? First, we ignore all non-Python files returned by the folder walk implemented in the linter's Rust core.

Second, we build an abstract syntax tree using ast, Python's built-in module. In short, this module allows us to walk through the nodes inside Python code and inspect them. Each node represents a construct from Python's grammar. For each node, we can check its type. For example, we can check whether a node is a constant, a variable, a function, and so on. We can also check the types of a function's arguments or its return type(s). This is used by tools like linters to inspect code and auto-update it. The same goes for jg-lint. If there's a syntax error, we just return an empty list. We let other tools like ruff to deal with such basic errors.

Third, we define a visit function to recursively walk the nodes of the abstract syntax tree. We track whether we're inside a function. If we find an import statement while we're inside a function, we report a violation by appending to the violations list. Each violation contains the file path, line number, column number, rule code, and rule message.

Last but not least, our check method returns all spotted violations. jg-lint walks through the files, provides their paths to the rule, and executes the rule.

Once we have our rule in place, we can add some inline import somewhere:

# src/example.py
def greet(name: str) -> str:
    import logging

    logging.info("Greeting %s", name)
    return f"Hello, {name}!"

After that, we can run jg-lint like this:

$ uv run jg-lint check src

It will produce a result like this:

src/example.py:3:5: JGL001 Inline import; move it to the module top level

Found 1 violation in 1 file.

Isn't that great? With that, we don't need to worry about inline imports anymore when reviewing the code. Check is automatically done by jg-lint.

Custom linter workflow for AI agents

While impressive, AI agents can also be easily distracted. That's why I strongly suggest you use make or just to define your workflow steps. Using them, you can easily define the agent's workflow step as Run code quality checks with make cq-agent. Inside the cq-agent definition, you can gather all commands that need to be executed - including jg-lint:

.PHONY: cq-agent
cq-agent:
    uv run ruff check --fix .
    uv run jg-lint check src

This way, the AI agent can run code quality checks and fix reported issues. Worst case, the CI/CD job for code quality will fail. That's still much better than hunting down inline imports during code review, isn't it? Similarly, you can implement rules for other things, such as "No IFs in tests" or "All emails in tests must be on our company's domain".

Note: The "no inline imports" rule is already implemented inside jg-lint, but it's such a great example that I'm showing it here as well.

Conclusion

We've talked a lot about AI agents. Nevertheless, all of that is relevant and useful even without AI agents. It's just that previously, producing the same amount of code took more time, so these issues weren't as visible. Therefore, I encourage you - instead of pushing for more code, invest in a tighter feedback loop, and you'll move much faster. Implement your code standards as custom lint rules - jg-lint makes that very simple. Feel free to implement everything on your own as well. The important bit is that AI agents can get feedback quickly and reliably.

Happy engineering!