We consider Banach algebras of infinite matrices defined in terms of a weight measuring the off-diagonal decay of the matrix entries. If a given matrix A is invertible as an operator on l^2 we analyze the decay of its inverse matrix entries in the case where the matrix algebra is not inverse closed in B(l^2), the Banach algebra of bounded operators on l^2. To this end we consider a condition on sequences of weights which extends the notion of GRS-condition. Finally we focus on the behavior of inverses of pseudodifferential operators whose Weyl symbols belong to weighted modulation spaces and the weights lack the GRS condition.